2024-03-29T10:40:26.457 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:40:26.458 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/orig.config.yaml 2024-03-29T10:40:26.493 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:40:26.574 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622658' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEtSkJut390BqVDfRKg4OAHfgPKyf9sIPkl6ioaxhFCEc4NbCyuccSf5xRUNVAYJB5c98YMumFwqGiBz6H09lQk= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOni4yEAR1WulQ0qK868whKpIbAuHVwVcRlsWDswMtE+hdMc1ichRhUwpUyRfbVf9GAPksxrWMZORAoAu2/LjRM= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:40:26.574 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:40:26.575 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:40:26.575 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:40:26.577 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:40:26.601 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:40:26.602 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:40:26.602 INFO:teuthology.packaging:ref: None 2024-03-29T10:40:26.602 INFO:teuthology.packaging:tag: None 2024-03-29T10:40:26.602 INFO:teuthology.packaging:branch: main 2024-03-29T10:40:26.602 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:40:26.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:40:26.955 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:40:26.957 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:40:26.963 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:40:26.964 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:40:26.980 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:40:26.995 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:40:27.001 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:40:27.025 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:35:09.256737', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEtSkJut390BqVDfRKg4OAHfgPKyf9sIPkl6ioaxhFCEc4NbCyuccSf5xRUNVAYJB5c98YMumFwqGiBz6H09lQk='} 2024-03-29T10:40:27.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:35:09.258771', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOni4yEAR1WulQ0qK868whKpIbAuHVwVcRlsWDswMtE+hdMc1ichRhUwpUyRfbVf9GAPksxrWMZORAoAu2/LjRM='} 2024-03-29T10:40:27.046 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:40:27.053 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:40:27.053 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:40:27.053 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:40:27.123 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb2e35c50d0>, signals=[15]) 2024-03-29T10:40:27.123 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:40:27.129 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:40:27.129 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T10:40:27.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:40:27.225 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T10:40:27.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:40:27.316 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:40:27.322 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-03-29T10:40:27.340 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-03-29T10:40:27.341 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:40:27.398 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-03-29T10:40:27.399 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:40:27.399 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:40:27.399 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:40:27.401 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:40:27.401 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:40:27.401 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:40:27.402 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-03-29T10:40:27.427 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-29T10:40:27.445 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-29T10:40:27.446 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-29T10:40:27.503 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-29T10:40:27.503 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:40:27.504 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:40:27.505 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-29T10:40:27.525 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:40:27.534 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:40:27.540 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:40:27.540 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:40:27.543 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:40:27.560 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:40:27.567 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:40:27.567 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:40:27.600 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:40:27.707 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:40:27.758 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:40:27.773 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:40:27.773 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:40:27.774 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:40:27.774 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:40:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:40:27.791 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-29T10:40:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:40:27.792 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-29T10:40:27.848 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:40:27.848 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-29T10:40:27.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-29T10:40:27.850 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:40:27.850 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-361.el9.x86_64 2024-03-29T10:40:27.850 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-03-29T10:40:29.804 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:40:30.018 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T10:40:31.171 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2024-03-29T10:40:31.468 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-29T10:40:32.228 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T10:40:36.713 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-03-29T10:40:36.850 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-03-29T10:40:41.005 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-29T10:40:41.284 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-03-29T10:40:43.247 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-03-29T10:40:44.684 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T10:40:45.351 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T10:40:46.936 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-29T10:40:48.095 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-29T10:40:49.673 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T10:40:51.896 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:40:51.896 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:40:51.989 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:40:51.999 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:40:51.999 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:40:51.999 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:40:51.999 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:40:51.999 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-29T10:40:52.000 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:40:52.001 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-29T10:40:52.001 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-29T10:40:52.001 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:40:52.140 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:40:52.716 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T10:40:53.108 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 30 MB/s | 34 MB 00:01 2024-03-29T10:40:53.144 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-29T10:40:53.491 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T10:40:53.494 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:53.494 INFO:teuthology.orchestra.run.smithi183.stdout:Total 51 MB/s | 76 MB 00:01 2024-03-29T10:40:53.494 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:40:53.854 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:40:53.855 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:40:54.707 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:40:54.708 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:40:55.435 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:40:55.435 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:40:55.526 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:40:55.530 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:40:55.530 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-03-29T10:40:55.531 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:40:55.532 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 76 M 2024-03-29T10:40:55.532 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 123 M 2024-03-29T10:40:55.532 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:40:55.712 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T10:40:56.171 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T10:40:56.663 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 30 MB/s | 34 MB 00:01 2024-03-29T10:40:56.998 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:40:57.104 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T10:40:57.107 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:57.108 INFO:teuthology.orchestra.run.smithi155.stdout:Total 49 MB/s | 76 MB 00:01 2024-03-29T10:40:57.108 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:40:57.473 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:40:57.473 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:40:57.582 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:40:57.674 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:40:58.330 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:40:58.330 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:41:00.427 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:00.569 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:41:01.254 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:41:01.337 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:04.120 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:04.246 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:05.277 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:05.318 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:08.975 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:09.012 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:10.010 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:13.735 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:39.498 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:41:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:41:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:39.933 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:39.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:41:39.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:39.933 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:41:40.302 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:41:40.626 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:40.731 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:40.731 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:40.731 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:41:40.731 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:41:40.731 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:41:40.732 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:41:40.732 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:41:40.732 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:40.813 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:40.813 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:40.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-29T10:41:40.852 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:41:40.853 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:41:40.854 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:41:40.855 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:41:40.855 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:41:40.855 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:41:41.514 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-29T10:41:42.866 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:41:42.898 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-29T10:41:42.900 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-29T10:41:42.918 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.okbbCvVaoo 2024-03-29T10:41:42.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.okbbCvVaoo 2024-03-29T10:41:42.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.okbbCvVaoo 2024-03-29T10:41:43.141 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.okbbCvVaoo is 6KB 2024-03-29T10:41:43.189 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.okbbCvVaoo 2024-03-29T10:41:43.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:41:43.222 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:43.222 INFO:teuthology.orchestra.run.smithi155.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:41:43.222 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:41:43.265 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:43.268 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:41:43.268 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:41:43.268 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:41:43.268 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:41:43.270 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:41:43.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-29T10:41:43.624 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:43.624 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:41:43.624 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:41:43.624 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:41:44.079 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:41:44.406 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:44.507 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:44.508 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:44.508 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:41:44.508 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:41:44.508 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:41:44.508 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:41:44.508 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:41:44.508 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:41:44.588 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:41:44.588 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:41:44.588 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-03-29T10:41:44.629 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-03-29T10:41:44.630 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:41:44.631 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:41:44.632 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:41:44.632 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:41:44.633 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:41:45.234 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-03-29T10:41:46.729 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:41:46.768 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-03-29T10:41:46.770 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-29T10:41:46.785 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.hP5Fy0ZQyD 2024-03-29T10:41:46.785 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hP5Fy0ZQyD 2024-03-29T10:41:46.853 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.hP5Fy0ZQyD 2024-03-29T10:41:46.995 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.hP5Fy0ZQyD is 6KB 2024-03-29T10:41:47.043 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.hP5Fy0ZQyD 2024-03-29T10:41:47.057 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:41:47.122 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:41:47.122 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:41:47.122 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:41:47.122 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:41:47.123 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:41:47.276 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-03-29T10:42:13.450 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:42:13.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:42:13.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:17.282 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:42:17.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T10:42:17.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:31.839 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-29T10:42:35.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-29T10:42:40.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:42:40.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:43.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-29T10:42:44.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T10:42:44.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:47.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-29T10:42:55.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:42:55.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:42:58.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-29T10:42:59.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T10:42:59.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:13.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:43:13.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:17.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-29T10:43:35.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:43:35.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:43:35.477 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-29T10:43:35.843 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:43:35.843 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:43:35.844 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-29T10:43:35.859 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:43:35.859 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:43:35.859 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:43:35.859 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:43:36.859 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:43:36.860 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:43:37.218 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-29T10:43:37.253 DEBUG:teuthology.parallel:result is None 2024-03-29T10:43:59.747 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:44:14.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T10:44:14.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:15.013 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-29T10:44:15.409 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T10:44:15.409 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:44:15.409 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-29T10:44:15.425 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:44:15.425 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:44:15.425 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:44:15.425 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:44:16.426 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:44:16.427 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:44:16.500 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-03-29T10:44:16.538 DEBUG:teuthology.parallel:result is None 2024-03-29T10:44:16.539 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:44:16.545 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:44:16.545 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:44:16.548 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:44:16.566 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:44:16.572 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:44:16.578 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:44:16.579 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:44:16.606 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:44:16.677 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:44:16.684 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:44:16.685 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:44:16.688 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:44:16.723 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:16.724 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:16.737 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:16.737 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:44:16.739 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:44:16.746 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:44:16.746 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:44:16.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:44:16.810 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:44:16.820 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:44:16.820 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:44:16.850 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:44:16.866 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:16.944 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:17.000 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:17.081 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:17.125 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:44:17.125 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:44:17.192 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:17.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:44:17.288 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:17.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:44:17.414 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:44:17.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:44:17.483 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-03-29T10:44:17.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-29T10:44:17.534 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:44:17.588 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:44:17.852 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:44:17.860 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:44:17.860 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:44:17.871 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:44:17.880 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-03-29T10:44:17.945 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-03-29T10:44:17.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-29T10:44:17.997 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-29T10:44:17.999 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:44:17.999 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-03-29T10:44:18.023 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-03-29T10:44:18.023 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-29T10:44:18.040 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-29T10:44:18.040 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi155.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:44:18.040 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:44:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:44:18.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:44:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:44:18.119 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:44:18.119 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:44:18.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:44:18.208 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:44:18.217 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:44:18.233 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:44:18.234 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:48:36.135 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-29T10:48:36.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T10:48:36.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:48:36.268 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-29T10:48:36.346 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-29T10:48:36.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:48:36.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:48:36.436 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-29T10:48:36.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-29T10:48:36.513 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:48:36.523 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:48:36.524 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:48:36.524 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:48:36.526 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:48:36.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:48:36.557 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:48:36.574 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:48:36.585 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:48:36.602 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:48:36.603 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-03-29T10:48:36.616 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-03-29T10:48:36.633 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:48:36.633 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-29T10:48:36.647 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-29T10:48:36.649 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:48:36.664 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:48:36.680 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:48:36.703 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:48:36.707 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:48:36.707 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-29T10:48:36.707 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.707 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.707 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.707 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.732 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:48:36.736 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:48:36.736 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-29T10:48:36.736 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.736 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.736 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.736 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:48:36.737 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:48:36.745 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:48:36.745 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:48:36.745 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:48:36.746 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:48:36.751 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:48:36.751 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:48:36.752 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:48:36.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:48:36.754 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:48:36.757 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:48:36.899 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:48:36.899 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:48:36.906 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:48:36.906 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:48:37.018 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:48:37.018 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:48:37.018 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:48:37.020 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:48:37.020 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:48:37.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:48:37.050 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:48:37.050 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:48:37.051 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:48:37.051 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:48:37.118 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:48:37.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:48:37.198 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:48:37.199 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:48:37.265 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-03-29T10:48:37.265 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-29T10:48:37.266 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-03-29T10:48:37.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-29T10:48:37.510 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-29T10:48:37.517 INFO:teuthology.orchestra.run.smithi155.stdout:74 files removed 2024-03-29T10:48:37.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-03-29T10:48:37.545 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-03-29T10:48:38.345 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 161 kB/s | 82 kB 00:00 2024-03-29T10:48:39.013 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-03-29T10:48:39.388 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-29T10:48:39.849 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:48:40.359 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 33 kB/s | 82 kB 00:02 2024-03-29T10:48:40.844 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-29T10:48:41.195 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-29T10:48:41.391 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-29T10:48:41.597 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T10:48:43.031 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-29T10:48:43.131 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-29T10:48:46.857 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-03-29T10:48:52.265 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.1 MB 00:01 2024-03-29T10:48:53.237 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 833 kB/s | 8.0 MB 00:09 2024-03-29T10:48:53.591 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 93 kB/s | 16 kB 00:00 2024-03-29T10:48:55.146 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-29T10:48:55.243 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T10:49:00.142 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T10:49:02.759 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:49:02.768 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:49:02.769 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:49:02.770 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:49:02.771 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout:Install 30 Packages 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-03-29T10:49:02.772 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:02.773 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-03-29T10:49:02.773 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:49:04.868 INFO:teuthology.orchestra.run.smithi183.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 10 MB/s | 20 MB 00:01 2024-03-29T10:49:04.943 INFO:teuthology.orchestra.run.smithi183.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-03-29T10:49:04.960 INFO:teuthology.orchestra.run.smithi183.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 274 kB/s | 25 kB 00:00 2024-03-29T10:49:05.027 INFO:teuthology.orchestra.run.smithi183.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T10:49:05.185 INFO:teuthology.orchestra.run.smithi183.stdout:(5/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.0 MB/s | 742 kB 00:00 2024-03-29T10:49:05.286 INFO:teuthology.orchestra.run.smithi183.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 450 kB/s | 45 kB 00:00 2024-03-29T10:49:05.444 INFO:teuthology.orchestra.run.smithi183.stdout:(7/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 9.7 MB 00:02 2024-03-29T10:49:05.461 INFO:teuthology.orchestra.run.smithi183.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 750 kB/s | 131 kB 00:00 2024-03-29T10:49:05.537 INFO:teuthology.orchestra.run.smithi183.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 10 MB/s | 5.1 MB 00:00 2024-03-29T10:49:05.562 INFO:teuthology.orchestra.run.smithi183.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.4 MB/s | 162 kB 00:00 2024-03-29T10:49:05.587 INFO:teuthology.orchestra.run.smithi183.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.5 MB/s | 320 kB 00:00 2024-03-29T10:49:05.612 INFO:teuthology.orchestra.run.smithi183.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T10:49:05.637 INFO:teuthology.orchestra.run.smithi183.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-29T10:49:05.780 INFO:teuthology.orchestra.run.smithi183.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 550 kB/s | 106 kB 00:00 2024-03-29T10:49:05.805 INFO:teuthology.orchestra.run.smithi183.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 828 kB/s | 159 kB 00:00 2024-03-29T10:49:05.821 INFO:teuthology.orchestra.run.smithi183.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 322 kB/s | 44 kB 00:00 2024-03-29T10:49:05.880 INFO:teuthology.orchestra.run.smithi183.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-29T10:49:05.897 INFO:teuthology.orchestra.run.smithi183.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 551 kB/s | 41 kB 00:00 2024-03-29T10:49:05.931 INFO:teuthology.orchestra.run.smithi183.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-03-29T10:49:05.955 INFO:teuthology.orchestra.run.smithi183.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T10:49:05.973 INFO:teuthology.orchestra.run.smithi183.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 466 kB/s | 35 kB 00:00 2024-03-29T10:49:05.989 INFO:teuthology.orchestra.run.smithi183.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T10:49:06.156 INFO:teuthology.orchestra.run.smithi183.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-03-29T10:49:06.290 INFO:teuthology.orchestra.run.smithi183.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-03-29T10:49:06.373 INFO:teuthology.orchestra.run.smithi183.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 740 kB/s | 309 kB 00:00 2024-03-29T10:49:06.440 INFO:teuthology.orchestra.run.smithi183.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 448 kB/s | 67 kB 00:00 2024-03-29T10:49:06.613 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 787 kB/s | 6.1 MB 00:07 2024-03-29T10:49:06.849 INFO:teuthology.orchestra.run.smithi183.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 469 kB/s | 192 kB 00:00 2024-03-29T10:49:06.916 INFO:teuthology.orchestra.run.smithi183.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-03-29T10:49:06.942 INFO:teuthology.orchestra.run.smithi183.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.4 MB/s | 838 kB 00:00 2024-03-29T10:49:07.200 INFO:teuthology.orchestra.run.smithi183.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-03-29T10:49:07.559 INFO:teuthology.orchestra.run.smithi183.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.3 MB/s | 3.4 MB 00:00 2024-03-29T10:49:07.725 INFO:teuthology.orchestra.run.smithi183.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 1.8 MB/s | 1.6 MB 00:00 2024-03-29T10:49:07.730 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:07.730 INFO:teuthology.orchestra.run.smithi183.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-29T10:49:07.730 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:49:07.768 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:49:07.768 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:49:07.997 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 16 kB 00:00 2024-03-29T10:49:08.239 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:49:08.240 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:49:08.982 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-29T10:49:09.746 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:49:09.820 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:49:09.854 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:49:09.880 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:49:09.903 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:09.920 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:10.077 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:10.457 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:49:10.506 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:10.598 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:10.689 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:49:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:49:10.868 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:49:10.938 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:10.960 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:11.024 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:11.059 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:49:11.168 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:49:11.201 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:49:11.242 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:49:11.274 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:49:11.303 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:49:11.336 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:49:11.657 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:49:11.682 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:49:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:49:12.291 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:49:12.327 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:12.397 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:12.465 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:49:12.495 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:49:13.874 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-29T10:49:14.612 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:14.658 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:15.220 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:15.253 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:15.338 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:15.338 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:49:15.338 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:49:15.338 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:15.379 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:16.504 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:49:16.511 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:49:16.512 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:49:16.513 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Install 30 Packages 2024-03-29T10:49:16.514 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-03-29T10:49:16.518 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:16.518 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 58 M 2024-03-29T10:49:16.519 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:49:17.441 INFO:teuthology.orchestra.run.smithi155.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 13 MB/s | 9.7 MB 00:00 2024-03-29T10:49:17.475 INFO:teuthology.orchestra.run.smithi155.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 759 kB/s | 25 kB 00:00 2024-03-29T10:49:17.541 INFO:teuthology.orchestra.run.smithi155.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 742 kB 00:00 2024-03-29T10:49:17.592 INFO:teuthology.orchestra.run.smithi155.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 9.7 MB/s | 498 kB 00:00 2024-03-29T10:49:17.884 INFO:teuthology.orchestra.run.smithi155.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-29T10:49:17.926 INFO:teuthology.orchestra.run.smithi155.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T10:49:17.976 INFO:teuthology.orchestra.run.smithi155.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T10:49:18.018 INFO:teuthology.orchestra.run.smithi155.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.8 MB/s | 162 kB 00:00 2024-03-29T10:49:18.076 INFO:teuthology.orchestra.run.smithi155.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 5.4 MB/s | 320 kB 00:00 2024-03-29T10:49:18.135 INFO:teuthology.orchestra.run.smithi155.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T10:49:18.177 INFO:teuthology.orchestra.run.smithi155.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 99 kB 00:00 2024-03-29T10:49:18.302 INFO:teuthology.orchestra.run.smithi155.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 847 kB/s | 106 kB 00:00 2024-03-29T10:49:18.345 INFO:teuthology.orchestra.run.smithi155.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T10:49:18.378 INFO:teuthology.orchestra.run.smithi155.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T10:49:18.428 INFO:teuthology.orchestra.run.smithi155.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T10:49:18.470 INFO:teuthology.orchestra.run.smithi155.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T10:49:18.504 INFO:teuthology.orchestra.run.smithi155.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T10:49:18.538 INFO:teuthology.orchestra.run.smithi155.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.4 MB/s | 220 kB 00:00 2024-03-29T10:49:18.571 INFO:teuthology.orchestra.run.smithi155.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T10:49:18.605 INFO:teuthology.orchestra.run.smithi155.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T10:49:18.789 INFO:teuthology.orchestra.run.smithi155.stdout:(21/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 2.6 MB/s | 5.3 MB 00:02 2024-03-29T10:49:18.872 INFO:teuthology.orchestra.run.smithi155.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-29T10:49:19.173 INFO:teuthology.orchestra.run.smithi155.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 87 kB/s | 26 kB 00:00 2024-03-29T10:49:19.239 INFO:teuthology.orchestra.run.smithi155.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-29T10:49:19.298 INFO:teuthology.orchestra.run.smithi155.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T10:49:19.574 INFO:teuthology.orchestra.run.smithi155.stdout:(26/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.9 MB/s | 20 MB 00:02 2024-03-29T10:49:19.607 INFO:teuthology.orchestra.run.smithi155.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-29T10:49:19.807 INFO:teuthology.orchestra.run.smithi155.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 822 kB/s | 192 kB 00:00 2024-03-29T10:49:19.866 INFO:teuthology.orchestra.run.smithi155.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-03-29T10:49:20.133 INFO:teuthology.orchestra.run.smithi155.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-03-29T10:49:20.641 INFO:teuthology.orchestra.run.smithi155.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.0 MB/s | 3.4 MB 00:00 2024-03-29T10:49:22.841 INFO:teuthology.orchestra.run.smithi155.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:04 2024-03-29T10:49:22.845 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:22.845 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.1 MB/s | 58 MB 00:06 2024-03-29T10:49:22.845 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:49:22.884 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:49:22.884 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:49:23.351 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:49:23.351 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:49:24.815 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:49:24.899 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:49:24.957 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:49:24.988 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:49:25.019 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:25.037 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:49:25.086 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:25.086 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-29T10:49:25.086 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-29T10:49:25.086 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-29T10:49:25.086 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-29T10:49:25.086 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-29T10:49:25.086 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-29T10:49:25.087 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-29T10:49:25.087 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:25.228 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:25.283 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:49:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:49:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:25.731 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:25.819 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:49:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:25.921 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:49:25.949 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:49:25.985 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:49:26.035 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:26.054 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:26.058 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:26.058 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:26.058 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:49:26.058 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:26.059 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:26.059 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:26.060 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:26.128 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:26.128 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:49:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:49:26.286 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:49:26.318 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:49:26.358 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:49:26.391 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:49:26.420 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:49:26.452 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:49:26.783 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:49:26.816 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:26.889 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:49:26.920 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:49:26.961 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:49:27.255 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:27.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:49:27.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:49:27.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:49:27.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:49:27.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:49:27.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:49:27.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:27.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:49:27.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:49:27.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:49:27.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:49:27.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:49:27.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:49:27.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:49:27.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:49:27.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:49:27.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:49:27.260 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:49:27.260 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:49:27.260 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:49:27.260 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:49:27.260 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:49:27.260 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:49:27.261 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:27.261 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:49:27.261 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:49:27.412 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:49:27.435 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:27.489 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:49:27.548 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:49:27.583 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.287 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:49:28.288 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:28.289 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:49:28.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-03-29T10:49:29.206 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:49:29.362 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:49:29.363 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:49:29.364 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:49:29.364 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:29.364 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:49:29.364 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:29.364 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-29T10:49:29.364 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:29.364 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 48 M 2024-03-29T10:49:29.365 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 204 M 2024-03-29T10:49:29.365 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:49:29.574 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:49:29.616 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T10:49:29.641 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T10:49:29.725 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:29.766 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:29.775 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T10:49:29.825 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 385 kB/s | 163 kB 00:00 2024-03-29T10:49:30.334 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:49:30.378 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:30.450 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:49:30.450 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:49:30.451 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:49:30.451 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:30.480 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:33.817 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 47 MB 00:04 2024-03-29T10:49:33.820 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:33.820 INFO:teuthology.orchestra.run.smithi183.stdout:Total 11 MB/s | 48 MB 00:04 2024-03-29T10:49:33.834 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:49:33.855 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:49:33.856 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:49:34.240 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:49:34.240 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:49:34.807 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:49:34.875 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:49:34.926 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:49:34.966 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:49:35.009 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:49:35.038 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:49:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:49:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:49:40.074 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:40.074 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-03-29T10:49:40.075 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-03-29T10:49:40.075 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-03-29T10:49:40.075 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-03-29T10:49:40.075 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-03-29T10:49:40.075 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-03-29T10:49:40.075 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-03-29T10:49:40.075 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:40.991 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:41.016 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:49:41.016 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:41.016 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:49:41.016 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:41.016 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:49:41.016 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:41.018 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:41.101 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:49:41.101 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:41.964 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:49:41.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:49:41.965 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:49:41.965 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:49:41.965 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:49:41.965 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:49:42.254 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:42.254 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:49:42.254 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:49:42.254 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:49:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:49:42.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:49:42.257 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:49:42.257 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:49:42.257 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:49:42.257 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:49:42.257 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:49:42.257 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:49:42.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:49:42.350 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:42.351 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:49:42.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-03-29T10:49:43.242 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:49:43.317 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:49:43.317 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:43.317 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-29T10:49:43.317 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.317 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:49:43.318 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:43.319 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:49:43.320 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:49:43.320 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:49:43.320 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:49:43.320 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:49:43.320 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:43.320 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:49:43.401 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:49:43.403 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-29T10:49:43.403 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:49:43.403 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-29T10:49:43.403 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:49:43.403 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:49:43.404 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:49:43.405 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:49:43.406 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-29T10:49:43.407 INFO:teuthology.orchestra.run.smithi183.stdout:Install 36 Packages 2024-03-29T10:49:43.407 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:43.407 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 30 M 2024-03-29T10:49:43.407 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 107 M 2024-03-29T10:49:43.407 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:49:43.591 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-03-29T10:49:44.148 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:31 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:49:44.301 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:49:44.301 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:49:44.301 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:49:44.302 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-29T10:49:44.303 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:49:44.303 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 48 M 2024-03-29T10:49:44.303 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 204 M 2024-03-29T10:49:44.303 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:49:44.370 INFO:teuthology.orchestra.run.smithi183.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-29T10:49:44.573 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T10:49:44.621 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-29T10:49:44.646 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T10:49:44.795 INFO:teuthology.orchestra.run.smithi183.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 2.1 MB 00:00 2024-03-29T10:49:44.829 INFO:teuthology.orchestra.run.smithi183.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-03-29T10:49:44.838 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-29T10:49:44.929 INFO:teuthology.orchestra.run.smithi183.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T10:49:45.063 INFO:teuthology.orchestra.run.smithi183.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T10:49:45.104 INFO:teuthology.orchestra.run.smithi183.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:49:45.105 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 237 kB/s | 163 kB 00:00 2024-03-29T10:49:45.129 INFO:teuthology.orchestra.run.smithi183.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T10:49:45.155 INFO:teuthology.orchestra.run.smithi183.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T10:49:45.190 INFO:teuthology.orchestra.run.smithi183.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:49:45.230 INFO:teuthology.orchestra.run.smithi183.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T10:49:45.306 INFO:teuthology.orchestra.run.smithi183.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:49:45.373 INFO:teuthology.orchestra.run.smithi183.stdout:(12/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 4.7 MB/s | 4.7 MB 00:01 2024-03-29T10:49:45.398 INFO:teuthology.orchestra.run.smithi183.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-29T10:49:45.423 INFO:teuthology.orchestra.run.smithi183.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:49:45.448 INFO:teuthology.orchestra.run.smithi183.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:49:45.549 INFO:teuthology.orchestra.run.smithi183.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-29T10:49:45.699 INFO:teuthology.orchestra.run.smithi183.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-29T10:49:45.741 INFO:teuthology.orchestra.run.smithi183.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-29T10:49:45.800 INFO:teuthology.orchestra.run.smithi183.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-29T10:49:45.992 INFO:teuthology.orchestra.run.smithi183.stdout:(20/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 14 MB/s | 17 MB 00:01 2024-03-29T10:49:46.017 INFO:teuthology.orchestra.run.smithi183.stdout:(21/36): python3-requests-2.25.1-8.el9.noarch.r 210 kB/s | 125 kB 00:00 2024-03-29T10:49:46.042 INFO:teuthology.orchestra.run.smithi183.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-29T10:49:46.067 INFO:teuthology.orchestra.run.smithi183.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-29T10:49:46.093 INFO:teuthology.orchestra.run.smithi183.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-29T10:49:46.118 INFO:teuthology.orchestra.run.smithi183.stdout:(25/36): python3-jaraco-classes-3.2.1-5.el9.noa 175 kB/s | 18 kB 00:00 2024-03-29T10:49:46.152 INFO:teuthology.orchestra.run.smithi183.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 231 kB/s | 19 kB 00:00 2024-03-29T10:49:46.177 INFO:teuthology.orchestra.run.smithi183.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-03-29T10:49:46.202 INFO:teuthology.orchestra.run.smithi183.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 550 kB/s | 46 kB 00:00 2024-03-29T10:49:46.230 INFO:teuthology.orchestra.run.smithi183.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T10:49:46.253 INFO:teuthology.orchestra.run.smithi183.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-03-29T10:49:46.278 INFO:teuthology.orchestra.run.smithi183.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.9 MB/s | 90 kB 00:00 2024-03-29T10:49:46.303 INFO:teuthology.orchestra.run.smithi183.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-29T10:49:46.329 INFO:teuthology.orchestra.run.smithi183.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-29T10:49:46.354 INFO:teuthology.orchestra.run.smithi183.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:49:46.371 INFO:teuthology.orchestra.run.smithi183.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 477 kB/s | 20 kB 00:00 2024-03-29T10:49:46.437 INFO:teuthology.orchestra.run.smithi183.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T10:49:46.440 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:46.440 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.8 MB/s | 30 MB 00:03 2024-03-29T10:49:46.464 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:49:46.507 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:49:46.507 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:49:46.912 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:49:46.913 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:49:48.262 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:49:48.323 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:49:48.366 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:49:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:49:48.470 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:49:48.516 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:49:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:49:48.640 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:49:48.718 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:49:48.760 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:49:48.831 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:49:48.885 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:49:48.924 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:49:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:49:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:49:49.218 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:49:49.270 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:49:49.300 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:49:49.335 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:49:49.365 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:49:49.441 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:49:49.476 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:49:49.509 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:49:49.549 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:49:49.584 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:49:49.627 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:49:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:49:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:49:49.829 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:49:49.966 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:49:50.093 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:49:50.249 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:49:50.280 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:49:50.305 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:49:50.305 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:50.305 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:49:50.305 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:49:50.306 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:49:50.306 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:51.588 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 6.6 MB/s | 47 MB 00:07 2024-03-29T10:49:51.591 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:51.591 INFO:teuthology.orchestra.run.smithi155.stdout:Total 6.6 MB/s | 48 MB 00:07 2024-03-29T10:49:51.606 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:49:51.628 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:49:51.628 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:49:51.741 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:49:51.764 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:49:51.765 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:51.765 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:49:51.765 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:49:51.765 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:49:51.765 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:52.017 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:49:52.018 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:49:52.206 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:49:52.229 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:49:52.229 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:52.230 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:49:52.230 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:49:52.230 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:49:52.230 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:52.432 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:49:52.455 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:49:52.455 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:49:52.455 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:49:52.455 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:49:52.455 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:49:52.455 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:52.521 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:49:52.594 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:49:52.663 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:49:52.712 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:49:52.762 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:49:52.796 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:49:52.816 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:49:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:49:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:49:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:49:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:49:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:49:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:49:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:49:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:49:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:49:55.201 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:49:55.202 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:49:55.203 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:49:55.204 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:55.205 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:49:55.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-03-29T10:49:56.124 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:49:56.223 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:49:56.275 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:49:56.276 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:49:56.277 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:49:56.340 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-03-29T10:49:56.909 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:49:57.061 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:49:57.062 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:57.062 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:49:57.062 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:57.062 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:49:57.062 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:49:57.062 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:57.062 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:49:57.063 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:49:57.063 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T10:49:57.063 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:57.063 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 747 k 2024-03-29T10:49:57.063 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 754 k 2024-03-29T10:49:57.063 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:49:57.492 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:49:57.573 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:49:57.899 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 895 kB/s | 747 kB 00:00 2024-03-29T10:49:57.899 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:49:57.900 INFO:teuthology.orchestra.run.smithi183.stdout:Total 893 kB/s | 747 kB 00:00 2024-03-29T10:49:57.900 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:49:57.906 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:49:57.906 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:49:57.912 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:49:57.913 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:49:57.987 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:49:58.417 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:58.513 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:58.887 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:59.173 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:49:59.173 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:59.173 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:49:59.173 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:49:59.173 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:49:59.173 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:49:59.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:49:59.965 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:00.059 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:50:00.059 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:50:00.059 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:50:00.059 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:50:00.059 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:50:00.059 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:50:00.118 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 145 k 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 436 k 2024-03-29T10:50:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:00.322 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 720 kB/s | 145 kB 00:00 2024-03-29T10:50:00.322 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:00.322 INFO:teuthology.orchestra.run.smithi183.stdout:Total 715 kB/s | 145 kB 00:00 2024-03-29T10:50:00.323 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:00.330 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:00.330 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:00.383 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:00.383 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:50:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:00.436 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:50:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:50:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:50:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:50:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:00.437 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:00.524 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:00.598 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:00.637 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:00.638 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:00.638 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:50:00.638 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:00.755 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-03-29T10:50:01.324 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:48 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:01.478 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:50:01.481 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:50:01.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:50:01.483 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout:========================================================================================= 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout:Install 36 Packages 2024-03-29T10:50:01.484 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:01.485 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 30 M 2024-03-29T10:50:01.485 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 107 M 2024-03-29T10:50:01.485 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:01.714 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:01.714 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:01.714 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:01.714 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:01.714 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:01.714 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:01.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-03-29T10:50:02.059 INFO:teuthology.orchestra.run.smithi155.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-03-29T10:50:02.243 INFO:teuthology.orchestra.run.smithi155.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.0 MB/s | 1.5 MB 00:00 2024-03-29T10:50:02.285 INFO:teuthology.orchestra.run.smithi155.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.0 MB/s | 2.1 MB 00:00 2024-03-29T10:50:02.344 INFO:teuthology.orchestra.run.smithi155.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.1 MB/s | 247 kB 00:00 2024-03-29T10:50:02.532 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:02.552 INFO:teuthology.orchestra.run.smithi155.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 897 kB/s | 187 kB 00:00 2024-03-29T10:50:02.603 INFO:teuthology.orchestra.run.smithi155.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T10:50:02.631 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:02.636 INFO:teuthology.orchestra.run.smithi155.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2024-03-29T10:50:02.670 INFO:teuthology.orchestra.run.smithi155.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:50:02.683 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:02.684 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:02.684 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:02.703 INFO:teuthology.orchestra.run.smithi155.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:50:02.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:50:02.768 INFO:teuthology.orchestra.run.smithi155.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.6 MB/s | 241 kB 00:00 2024-03-29T10:50:02.838 INFO:teuthology.orchestra.run.smithi155.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T10:50:02.871 INFO:teuthology.orchestra.run.smithi155.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:50:02.905 INFO:teuthology.orchestra.run.smithi155.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:50:02.955 INFO:teuthology.orchestra.run.smithi155.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:50:03.030 INFO:teuthology.orchestra.run.smithi155.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 4.8 MB/s | 4.7 MB 00:00 2024-03-29T10:50:03.072 INFO:teuthology.orchestra.run.smithi155.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-29T10:50:03.140 INFO:teuthology.orchestra.run.smithi155.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-03-29T10:50:03.306 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:03.340 INFO:teuthology.orchestra.run.smithi155.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T10:50:03.407 INFO:teuthology.orchestra.run.smithi155.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 453 kB/s | 151 kB 00:00 2024-03-29T10:50:03.432 INFO:teuthology.orchestra.run.smithi155.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 149 kB/s | 43 kB 00:00 2024-03-29T10:50:03.458 INFO:teuthology.orchestra.run.smithi155.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-03-29T10:50:03.458 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:03.459 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-29T10:50:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:50:03.459 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-29T10:50:03.459 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:03.459 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:50:03.459 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:03.460 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-29T10:50:03.461 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-29T10:50:03.461 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:03.461 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.9 M 2024-03-29T10:50:03.461 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-29T10:50:03.461 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:03.482 INFO:teuthology.orchestra.run.smithi155.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-03-29T10:50:03.508 INFO:teuthology.orchestra.run.smithi155.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-03-29T10:50:03.533 INFO:teuthology.orchestra.run.smithi155.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 237 kB/s | 18 kB 00:00 2024-03-29T10:50:03.559 INFO:teuthology.orchestra.run.smithi155.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T10:50:03.584 INFO:teuthology.orchestra.run.smithi155.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-29T10:50:03.609 INFO:teuthology.orchestra.run.smithi155.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-29T10:50:03.634 INFO:teuthology.orchestra.run.smithi155.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 616 kB/s | 46 kB 00:00 2024-03-29T10:50:03.660 INFO:teuthology.orchestra.run.smithi155.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T10:50:03.743 INFO:teuthology.orchestra.run.smithi155.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-29T10:50:03.768 INFO:teuthology.orchestra.run.smithi155.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-03-29T10:50:03.810 INFO:teuthology.orchestra.run.smithi155.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 598 kB/s | 90 kB 00:00 2024-03-29T10:50:03.827 INFO:teuthology.orchestra.run.smithi155.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 429 kB/s | 36 kB 00:00 2024-03-29T10:50:03.861 INFO:teuthology.orchestra.run.smithi155.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T10:50:03.886 INFO:teuthology.orchestra.run.smithi155.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-29T10:50:03.911 INFO:teuthology.orchestra.run.smithi155.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-29T10:50:03.917 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:03.917 INFO:teuthology.orchestra.run.smithi155.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-29T10:50:03.942 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:03.986 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:03.986 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:04.026 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 61 kB/s | 15 kB 00:00 2024-03-29T10:50:04.051 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 113 kB/s | 31 kB 00:00 2024-03-29T10:50:04.193 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T10:50:04.227 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T10:50:04.269 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.6 MB/s | 1.8 MB 00:00 2024-03-29T10:50:04.319 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 387 kB/s | 19 kB 00:00 2024-03-29T10:50:04.386 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:04.386 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:04.444 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-03-29T10:50:04.470 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-03-29T10:50:04.495 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-29T10:50:04.520 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 611 kB/s | 31 kB 00:00 2024-03-29T10:50:04.546 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-29T10:50:04.587 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.1 MB/s | 863 kB 00:00 2024-03-29T10:50:04.591 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:04.591 INFO:teuthology.orchestra.run.smithi183.stdout:Total 6.1 MB/s | 6.9 MB 00:01 2024-03-29T10:50:04.609 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:04.626 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:04.627 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:04.807 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:04.807 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:05.315 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:05.379 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:50:05.415 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:50:05.461 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:50:05.492 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:50:05.594 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:50:05.717 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:50:05.719 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:05.781 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:50:05.802 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:50:05.820 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:50:05.855 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:50:05.858 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:50:05.883 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:05.915 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:50:05.929 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:50:05.982 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:50:06.023 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:50:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:50:06.160 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:50:06.182 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:06.186 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:50:06.226 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:50:06.298 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:50:06.351 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:50:06.390 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:50:06.439 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:50:06.496 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:50:06.679 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:50:06.737 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:50:06.775 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:50:06.810 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:50:06.840 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:50:06.917 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:50:06.948 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:50:06.949 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:50:06.949 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:06.949 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:50:06.950 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:50:06.950 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:50:06.992 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:50:07.032 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:50:07.067 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:50:07.102 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:50:07.191 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:50:07.234 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:50:07.305 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:50:07.314 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:50:07.314 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:07.314 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:07.314 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:50:07.314 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:07.314 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:07.314 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:07.315 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:50:07.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:50:07.569 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:50:07.726 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:50:07.754 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:50:07.781 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:50:07.781 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:07.781 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:50:07.781 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:50:07.781 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:50:07.781 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:08.117 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:08.270 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:50:08.271 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-29T10:50:08.272 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:08.273 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2024-03-29T10:50:08.273 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2024-03-29T10:50:08.273 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:08.577 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 208 kB/s | 32 kB 00:00 2024-03-29T10:50:08.611 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 521 kB/s | 17 kB 00:00 2024-03-29T10:50:08.653 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 34 kB 00:00 2024-03-29T10:50:08.712 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T10:50:08.928 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-03-29T10:50:08.995 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T10:50:09.021 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:50:09.121 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 11 MB/s | 7.4 MB 00:00 2024-03-29T10:50:09.222 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:50:09.250 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:50:09.250 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:09.250 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:50:09.250 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:50:09.251 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:50:09.251 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:09.505 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-29T10:50:09.663 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-29T10:50:09.689 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:50:09.712 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:50:09.713 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:09.713 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:50:09.713 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:50:09.713 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:50:09.713 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:09.805 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 287 kB/s | 195 kB 00:00 2024-03-29T10:50:09.923 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:50:09.947 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:50:09.947 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:09.947 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:50:09.948 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:50:09.948 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:50:09.948 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:10.005 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:50:10.155 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 708 kB/s | 803 kB 00:01 2024-03-29T10:50:10.157 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:10.158 INFO:teuthology.orchestra.run.smithi183.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-29T10:50:10.202 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:10.227 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:10.227 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:10.589 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:10.589 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:11.303 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:11.369 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:50:11.401 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:50:11.493 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:11.547 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:50:11.722 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:11.780 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:50:11.781 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:50:11.782 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:50:11.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:50:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:50:12.493 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:50:12.598 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:50:12.907 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:50:12.907 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:12.907 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:50:12.908 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:50:12.909 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:50:12.910 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:50:12.910 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:50:12.910 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:50:12.910 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:50:12.910 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:50:12.911 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:50:12.912 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:50:12.912 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:50:12.912 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:12.912 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:13.274 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-03-29T10:50:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:50:13.843 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:13.941 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:13.994 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:13.995 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:50:13.995 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:14.067 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-03-29T10:50:14.632 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:14.783 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 747 k 2024-03-29T10:50:14.784 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 754 k 2024-03-29T10:50:14.785 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:14.860 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:50:14.886 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:15.069 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T10:50:15.070 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:15.070 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T10:50:15.071 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:15.079 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:15.079 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:15.081 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:15.082 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:15.156 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:15.613 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:15.708 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:16.095 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:16.395 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:50:16.395 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:16.395 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:16.395 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:16.395 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:16.395 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:16.668 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:50:17.234 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:17.388 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:17.388 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:17.388 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T10:50:17.389 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:17.390 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 145 k 2024-03-29T10:50:17.390 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 436 k 2024-03-29T10:50:17.390 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:17.397 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:17.398 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:50:17.398 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:50:17.398 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:17.398 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:50:17.398 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:17.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:17.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:50:17.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:50:17.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:50:17.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:50:17.399 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:50:17.752 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 402 kB/s | 145 kB 00:00 2024-03-29T10:50:17.752 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:17.752 INFO:teuthology.orchestra.run.smithi155.stdout:Total 400 kB/s | 145 kB 00:00 2024-03-29T10:50:17.753 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:17.760 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:17.760 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:17.813 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:17.813 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:50:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:17.825 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:17.961 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:18.035 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:18.060 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:18.061 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:18.061 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:50:18.061 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:18.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:50:18.708 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:18.858 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:18.859 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T10:50:18.859 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:50:18.859 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T10:50:18.859 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:18.859 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:50:18.859 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:50:18.860 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:50:18.861 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:50:18.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:18.861 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:18.861 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T10:50:18.861 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-03-29T10:50:18.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:18.862 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-03-29T10:50:18.862 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-29T10:50:18.862 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:19.005 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:50:19.005 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:19.005 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:19.005 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:19.005 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:19.006 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:19.062 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 24 kB 00:00 2024-03-29T10:50:19.087 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-29T10:50:19.137 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:50:19.163 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:50:19.188 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:50:19.213 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 852 kB/s | 43 kB 00:00 2024-03-29T10:50:19.238 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-03-29T10:50:19.238 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 174 kB/s | 52 kB 00:00 2024-03-29T10:50:19.489 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-29T10:50:19.514 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-29T10:50:19.698 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-29T10:50:19.723 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 419 kB/s | 203 kB 00:00 2024-03-29T10:50:19.812 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:19.848 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-03-29T10:50:19.898 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 448 kB/s | 90 kB 00:00 2024-03-29T10:50:19.902 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:19.902 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-29T10:50:19.902 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:19.911 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:19.915 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:19.915 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:19.965 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:19.966 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:50:19.966 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:20.006 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:20.006 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:20.041 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:50:20.524 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:20.608 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:50:20.749 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:50:20.761 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:20.762 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-29T10:50:20.763 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:20.763 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-29T10:50:20.763 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:20.763 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:50:20.763 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:50:20.764 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:50:20.764 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:50:20.764 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:50:20.764 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:50:20.764 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:50:20.764 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================== 2024-03-29T10:50:20.765 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-03-29T10:50:20.766 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:20.767 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.9 M 2024-03-29T10:50:20.767 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-03-29T10:50:20.767 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:20.803 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:50:20.834 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:50:20.875 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:50:20.967 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:50:21.076 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:50:21.081 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 81 kB/s | 15 kB 00:00 2024-03-29T10:50:21.126 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:50:21.173 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 112 kB/s | 31 kB 00:00 2024-03-29T10:50:21.232 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-29T10:50:21.257 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T10:50:21.307 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.3 MB/s | 1.8 MB 00:00 2024-03-29T10:50:21.516 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-29T10:50:21.566 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-29T10:50:21.750 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T10:50:21.775 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 780 kB/s | 144 kB 00:00 2024-03-29T10:50:21.834 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 367 kB/s | 31 kB 00:00 2024-03-29T10:50:21.839 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:50:21.890 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:50:21.892 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T10:50:21.923 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:50:22.000 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:50:22.023 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:22.109 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-03-29T10:50:22.114 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:22.115 INFO:teuthology.orchestra.run.smithi155.stdout:Total 5.1 MB/s | 6.9 MB 00:01 2024-03-29T10:50:22.132 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:22.149 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:22.150 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:22.329 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:22.330 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:22.724 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:22.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:50:22.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:50:22.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:50:22.851 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:22.918 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:50:22.962 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:50:23.008 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:50:23.047 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:50:23.094 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:50:23.094 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:23.094 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:23.094 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:23.094 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:50:23.094 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:50:23.094 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:23.095 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:23.148 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:50:23.281 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:50:23.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:50:23.384 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:50:23.427 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:50:23.456 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:23.488 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:50:23.733 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:50:23.771 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:23.918 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:24.071 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:24.072 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T10:50:24.072 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:50:24.072 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T10:50:24.072 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:24.072 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:50:24.072 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-29T10:50:24.073 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:24.074 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2024-03-29T10:50:24.074 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-03-29T10:50:24.074 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:24.516 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 548 kB/s | 138 kB 00:00 2024-03-29T10:50:24.567 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:50:24.567 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:50:24.567 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:50:24.567 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:50:24.567 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:50:24.568 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:50:24.568 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:50:24.568 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:50:24.568 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:50:24.568 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:50:24.568 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:50:24.568 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:50:24.658 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-29T10:50:24.808 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T10:50:24.834 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2024-03-29T10:50:24.859 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 417 kB/s | 248 kB 00:00 2024-03-29T10:50:24.884 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 676 kB/s | 51 kB 00:00 2024-03-29T10:50:24.885 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:24.885 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.4 MB/s | 6.8 MB 00:00 2024-03-29T10:50:24.899 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:24.909 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:24.909 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:24.917 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:50:24.917 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:24.917 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:24.917 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:24.918 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:24.998 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:24.999 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:25.198 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:50:25.297 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:25.459 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:50:25.529 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:50:25.759 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:12 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:25.913 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:25.914 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-29T10:50:25.914 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:25.914 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-29T10:50:25.914 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:25.914 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:50:25.914 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:50:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:50:25.916 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:50:25.916 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:25.916 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:25.916 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-29T10:50:25.916 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-03-29T10:50:25.916 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:25.916 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 37 M 2024-03-29T10:50:25.917 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 209 M 2024-03-29T10:50:25.917 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:26.050 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:50:26.132 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:50:26.194 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:50:26.233 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:26.292 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 225 kB/s | 32 kB 00:00 2024-03-29T10:50:26.325 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 519 kB/s | 17 kB 00:00 2024-03-29T10:50:26.367 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 860 kB/s | 34 kB 00:00 2024-03-29T10:50:26.418 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:50:26.609 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:50:26.676 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:50:26.701 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:50:26.827 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.4 MB/s | 803 kB 00:00 2024-03-29T10:50:26.999 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:26.999 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:50:26.999 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:50:26.999 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:50:26.999 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:50:26.999 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:50:27.044 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T10:50:27.070 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 815 kB/s | 195 kB 00:00 2024-03-29T10:50:27.318 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:50:27.318 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:27.318 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:27.319 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:27.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-03-29T10:50:27.595 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 446 kB/s | 245 kB 00:00 2024-03-29T10:50:27.770 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.6 MB/s | 7.4 MB 00:01 2024-03-29T10:50:27.772 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:27.773 INFO:teuthology.orchestra.run.smithi155.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-29T10:50:27.815 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:27.839 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:27.839 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:28.128 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:28.193 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:28.193 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:28.278 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:28.278 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:28.278 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:50:28.278 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:28.278 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:28.278 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:50:28.278 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:50:28.279 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:50:28.279 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:28.279 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:28.279 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:28.279 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-29T10:50:28.279 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:28.279 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 964 k 2024-03-29T10:50:28.280 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.8 M 2024-03-29T10:50:28.280 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:28.396 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 677 kB/s | 79 kB 00:00 2024-03-29T10:50:28.891 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:28.959 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:50:28.999 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:50:29.093 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:29.144 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:50:29.288 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 878 kB/s | 885 kB 00:01 2024-03-29T10:50:29.289 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:29.289 INFO:teuthology.orchestra.run.smithi183.stdout:Total 954 kB/s | 964 kB 00:01 2024-03-29T10:50:29.290 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:29.311 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:29.311 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:29.320 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:29.351 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:29.388 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:29.388 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:29.426 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:50:29.573 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:29.706 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:50:29.781 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:30.078 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:50:30.189 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:50:30.536 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:30.536 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:50:30.852 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:50:30.852 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:30.852 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:30.852 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:50:30.852 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:30.852 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:31.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-03-29T10:50:31.467 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:50:31.695 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:31.846 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:31.846 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-29T10:50:31.847 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:31.848 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 332 k 2024-03-29T10:50:31.848 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.5 M 2024-03-29T10:50:31.848 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:31.967 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 580 kB/s | 69 kB 00:00 2024-03-29T10:50:32.434 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 450 kB/s | 264 kB 00:00 2024-03-29T10:50:32.437 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:32.438 INFO:teuthology.orchestra.run.smithi183.stdout:Total 565 kB/s | 332 kB 00:00 2024-03-29T10:50:32.438 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:32.444 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:32.444 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:32.481 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:32.481 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:32.498 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:50:32.527 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:32.630 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:32.791 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:50:32.820 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:32.836 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:32.836 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:32.837 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:50:32.837 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:33.615 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:50:33.960 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:50:33.960 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:33.960 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:33.960 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:33.960 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:50:33.961 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:33.961 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:34.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-03-29T10:50:34.870 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:35.023 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:35.023 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:35.023 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T10:50:35.024 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:35.025 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 127 k 2024-03-29T10:50:35.025 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 456 k 2024-03-29T10:50:35.025 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:50:35.081 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:50:35.082 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:50:35.082 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:50:35.082 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:50:35.416 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 324 kB/s | 127 kB 00:00 2024-03-29T10:50:35.416 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:35.416 INFO:teuthology.orchestra.run.smithi183.stdout:Total 323 kB/s | 127 kB 00:00 2024-03-29T10:50:35.416 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:35.422 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:35.422 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:35.479 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:35.479 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:35.577 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:50:35.578 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:35.579 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:50:35.579 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:50:35.579 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:35.579 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:35.625 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:35.968 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:50:36.181 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:36.483 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:36.483 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:36.483 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:36.484 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:36.484 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:36.484 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:36.542 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:36.693 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:50:36.695 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-03-29T10:50:36.696 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:36.697 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.0 M 2024-03-29T10:50:36.697 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-03-29T10:50:36.697 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:36.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-03-29T10:50:36.962 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 164 kB/s | 18 kB 00:00 2024-03-29T10:50:36.978 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 24 kB 00:00 2024-03-29T10:50:37.045 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-29T10:50:37.071 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-29T10:50:37.096 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:50:37.113 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T10:50:37.188 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-29T10:50:37.205 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2024-03-29T10:50:37.231 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 138 kB/s | 52 kB 00:00 2024-03-29T10:50:37.294 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:37.306 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.7 MB/s | 203 kB 00:00 2024-03-29T10:50:37.331 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-29T10:50:37.356 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-29T10:50:37.390 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-03-29T10:50:37.391 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:37.392 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2024-03-29T10:50:37.392 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:37.396 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:37.405 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:37.405 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:37.451 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:37.452 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:37.452 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:37.495 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:37.495 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:37.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-03-29T10:50:38.023 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:38.095 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:38.185 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:50:38.241 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:50:38.247 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:38.247 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:38.247 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T10:50:38.247 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:38.248 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 31 k 2024-03-29T10:50:38.249 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 140 k 2024-03-29T10:50:38.249 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:38.304 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:50:38.334 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:50:38.375 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:50:38.477 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:50:38.586 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:50:38.643 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:50:38.693 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 70 kB/s | 31 kB 00:00 2024-03-29T10:50:38.694 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:38.694 INFO:teuthology.orchestra.run.smithi183.stdout:Total 70 kB/s | 31 kB 00:00 2024-03-29T10:50:38.694 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:38.698 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:38.698 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:38.729 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:38.729 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:38.865 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:38.967 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:39.340 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:50:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:50:39.441 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:50:39.490 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:50:39.511 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:39.608 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:39.608 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:39.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:50:40.335 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:50:40.336 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:50:40.336 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:50:40.336 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:50:40.336 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:50:40.339 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:40.437 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:40.488 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:40.489 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:40.489 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:40.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:50:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:50:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:50:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:50:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:50:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:50:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:50:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:40.700 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:40.959 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:50:41.107 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:41.204 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:41.254 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:41.255 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:41.255 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:41.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-03-29T10:50:41.525 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:41.678 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:41.678 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T10:50:41.678 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:41.678 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.8 M 2024-03-29T10:50:41.679 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-03-29T10:50:41.680 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:41.870 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:41.969 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.1 MB/s | 248 kB 00:00 2024-03-29T10:50:42.022 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:42.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-03-29T10:50:42.302 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T10:50:42.428 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-29T10:50:42.461 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-03-29T10:50:42.486 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 872 kB/s | 51 kB 00:00 2024-03-29T10:50:42.553 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 210 kB/s | 138 kB 00:00 2024-03-29T10:50:42.555 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:42.555 INFO:teuthology.orchestra.run.smithi155.stdout:Total 7.8 MB/s | 6.8 MB 00:00 2024-03-29T10:50:42.568 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:42.579 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:42.579 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:42.644 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:42.668 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:42.669 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:42.743 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:42.795 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:42.796 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:42.796 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:42.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-03-29T10:50:42.978 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:43.154 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:50:43.205 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:50:43.425 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:43.524 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:43.577 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:43.578 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:43.578 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:43.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-03-29T10:50:43.728 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:50:43.808 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:50:43.870 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:50:43.893 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:44.201 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:44.299 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:44.351 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:44.352 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-29T10:50:44.352 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:44.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-03-29T10:50:44.710 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:50:44.710 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:50:44.710 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:50:44.710 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:50:44.710 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:50:44.710 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:50:44.967 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:45.013 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:45.014 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T10:50:45.119 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:45.119 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 85 k 2024-03-29T10:50:45.119 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 231 k 2024-03-29T10:50:45.119 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:45.273 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-03-29T10:50:45.663 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 157 kB/s | 85 kB 00:00 2024-03-29T10:50:45.664 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:45.664 INFO:teuthology.orchestra.run.smithi183.stdout:Total 156 kB/s | 85 kB 00:00 2024-03-29T10:50:45.664 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:45.671 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:45.671 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:45.727 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:45.727 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:45.847 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:45.863 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:45.969 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:45.997 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:45.997 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:45.997 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:45.997 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:45.997 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-29T10:50:45.998 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:45.999 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 964 k 2024-03-29T10:50:45.999 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.8 M 2024-03-29T10:50:45.999 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:46.138 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 564 kB/s | 79 kB 00:00 2024-03-29T10:50:46.342 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:46.634 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:46.635 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:46.806 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 885 kB 00:00 2024-03-29T10:50:46.806 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:46.807 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-03-29T10:50:46.807 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:46.828 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:46.828 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:46.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-03-29T10:50:46.902 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:46.902 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:47.094 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:50:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:47.466 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:47.638 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:47.639 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:47.639 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-29T10:50:47.639 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:47.639 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:47.639 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:50:47.639 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:47.640 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:47.640 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:47.640 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T10:50:47.640 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:47.640 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.1 M 2024-03-29T10:50:47.640 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 13 M 2024-03-29T10:50:47.640 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:47.990 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:50:47.990 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:50:48.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:50:48.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:48.266 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:48.266 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:50:48.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:48.266 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:48.589 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-volume 2024-03-29T10:50:48.766 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.7 MB/s | 3.1 MB 00:01 2024-03-29T10:50:48.772 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:48.772 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.7 MB/s | 3.1 MB 00:01 2024-03-29T10:50:48.772 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:48.777 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:48.777 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:48.863 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:48.863 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:49.167 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:49.290 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:49.326 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:49.326 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:49.326 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-29T10:50:49.326 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:49.326 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-29T10:50:49.327 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:49.328 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 332 k 2024-03-29T10:50:49.328 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.5 M 2024-03-29T10:50:49.328 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:49.343 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:49.368 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:49.368 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:49.368 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:50:49.369 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:50:49.369 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:50:49.369 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:49.453 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 547 kB/s | 69 kB 00:00 2024-03-29T10:50:50.020 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 381 kB/s | 264 kB 00:00 2024-03-29T10:50:50.021 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:50.022 INFO:teuthology.orchestra.run.smithi155.stdout:Total 478 kB/s | 332 kB 00:00 2024-03-29T10:50:50.022 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:50.030 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:50.030 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:50.071 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:50.071 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:50.219 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:50.402 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:50:50.439 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:50.455 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:50:50.455 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:50:50.455 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:50:50.456 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:50.539 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:50.539 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:50.539 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:50.539 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:50.539 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:50.539 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:50.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-03-29T10:50:51.178 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:50:51.352 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:49:00 AM UTC. 2024-03-29T10:50:51.502 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-29T10:50:51.503 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-29T10:50:51.503 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:51.503 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-29T10:50:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:50:51.503 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:51.504 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-29T10:50:51.504 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-29T10:50:51.504 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-29T10:50:51.504 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:51.504 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 171 k 2024-03-29T10:50:51.504 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 494 k 2024-03-29T10:50:51.504 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-29T10:50:51.513 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:50:51.513 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:51.513 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:51.513 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:50:51.513 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:50:51.513 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:51.514 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:51.868 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-03-29T10:50:51.987 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 355 kB/s | 171 kB 00:00 2024-03-29T10:50:51.988 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:51.988 INFO:teuthology.orchestra.run.smithi183.stdout:Total 354 kB/s | 171 kB 00:00 2024-03-29T10:50:51.988 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-29T10:50:51.995 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-29T10:50:51.995 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-29T10:50:52.052 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-29T10:50:52.052 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-29T10:50:52.196 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-29T10:50:52.301 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:52.489 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:52.639 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:52.640 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:52.640 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T10:50:52.640 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:52.640 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:52.640 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:50:52.640 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:52.640 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:52.641 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:52.641 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T10:50:52.641 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:52.641 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 127 k 2024-03-29T10:50:52.641 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 456 k 2024-03-29T10:50:52.641 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:53.021 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:53.021 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:53.021 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-29T10:50:53.022 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:53.022 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:50:53.022 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-29T10:50:53.111 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 269 kB/s | 127 kB 00:00 2024-03-29T10:50:53.112 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:53.112 INFO:teuthology.orchestra.run.smithi155.stdout:Total 268 kB/s | 127 kB 00:00 2024-03-29T10:50:53.112 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:53.118 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:53.118 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:53.174 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:53.174 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:53.255 DEBUG:teuthology.parallel:result is None 2024-03-29T10:50:53.313 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:53.410 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:54.213 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:54.213 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:54.214 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:54.214 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:54.214 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:54.214 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:54.447 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-03-29T10:50:55.027 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:55.130 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:55.184 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:55.185 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:50:55.185 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:55.249 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-03-29T10:50:55.820 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:55.975 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:55.976 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 31 k 2024-03-29T10:50:55.977 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 140 k 2024-03-29T10:50:55.977 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:50:56.473 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 63 kB/s | 31 kB 00:00 2024-03-29T10:50:56.474 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:50:56.474 INFO:teuthology.orchestra.run.smithi155.stdout:Total 63 kB/s | 31 kB 00:00 2024-03-29T10:50:56.474 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:50:56.477 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:50:56.477 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:50:56.508 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:50:56.509 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:50:56.633 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:50:56.729 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:57.081 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:57.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:50:57.361 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:57.361 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:50:57.361 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:50:57.361 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:50:57.361 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:57.535 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-03-29T10:50:58.096 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:58.194 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:58.245 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:58.247 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:50:58.247 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:58.295 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-03-29T10:50:58.858 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:58.956 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:59.007 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:59.008 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:50:59.008 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:59.050 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-03-29T10:50:59.616 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:50:59.714 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:50:59.766 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:50:59.767 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:50:59.767 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:50:59.816 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-03-29T10:51:00.381 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:51:00.479 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:00.531 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:51:00.532 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:51:00.532 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:51:00.575 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-03-29T10:51:01.146 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:51:01.245 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:01.296 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:51:01.297 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:51:01.297 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:51:01.342 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-03-29T10:51:01.908 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:51:02.006 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:51:02.058 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:51:02.059 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-29T10:51:02.059 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:51:02.106 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-03-29T10:51:02.677 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:51:02.836 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T10:51:02.837 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:02.838 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 85 k 2024-03-29T10:51:02.838 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 231 k 2024-03-29T10:51:02.838 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:51:03.402 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 151 kB/s | 85 kB 00:00 2024-03-29T10:51:03.403 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:03.404 INFO:teuthology.orchestra.run.smithi155.stdout:Total 151 kB/s | 85 kB 00:00 2024-03-29T10:51:03.404 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:51:03.416 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:51:03.416 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:51:03.479 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:51:03.480 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:51:03.631 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:51:03.726 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:04.203 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:04.510 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:04.510 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:04.510 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:51:04.510 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:04.510 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:04.510 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:51:04.729 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-03-29T10:51:05.312 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:51:05.466 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.1 M 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 13 M 2024-03-29T10:51:05.467 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:51:07.189 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.8 MB/s | 3.1 MB 00:01 2024-03-29T10:51:07.190 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:07.190 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.8 MB/s | 3.1 MB 00:01 2024-03-29T10:51:07.191 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:51:07.201 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:51:07.201 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:51:07.278 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:51:07.278 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:51:07.712 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:51:07.773 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:07.797 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:07.798 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:51:07.798 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:51:07.798 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:51:07.798 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:51:07.798 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:08.922 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:08.922 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:08.922 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:51:08.922 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:08.922 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:08.922 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:51:09.195 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-03-29T10:51:09.771 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:49:13 AM UTC. 2024-03-29T10:51:09.919 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-29T10:51:09.920 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-29T10:51:09.920 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:09.920 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 171 k 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 494 k 2024-03-29T10:51:09.921 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-29T10:51:10.377 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 376 kB/s | 171 kB 00:00 2024-03-29T10:51:10.377 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:10.377 INFO:teuthology.orchestra.run.smithi155.stdout:Total 375 kB/s | 171 kB 00:00 2024-03-29T10:51:10.377 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-29T10:51:10.386 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-29T10:51:10.386 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-29T10:51:10.442 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-29T10:51:10.442 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-29T10:51:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-29T10:51:10.700 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:11.417 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:51:11.418 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:11.418 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-29T10:51:11.418 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:51:11.418 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:11.418 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-29T10:51:11.682 DEBUG:teuthology.parallel:result is None 2024-03-29T10:51:11.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:51:11.921 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:51:11.943 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:11.944 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:11.944 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:51:11.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:51:12.094 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:51:12.119 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:12.119 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:51:12.119 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:51:12.121 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:51:12.121 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:12.121 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:51:12.151 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:12.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:51:12.191 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:51:12.191 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:12.191 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:51:12.219 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:51:12.287 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:12.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:51:12.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:51:12.388 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:51:12.389 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:12.389 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:51:12.419 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:51:12.486 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:12.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:51:12.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:51:12.587 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:51:12.587 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:12.587 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:51:12.615 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:51:12.682 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:12.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:51:12.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:51:12.782 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:51:12.871 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:51:12.871 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2024-03-29T10:51:12.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-03-29T10:51:12.903 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:51:12.903 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:51:12.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:51:12.972 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:51:12.972 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:51:13.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:51:13.052 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:51:13.052 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:51:13.052 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622658', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEtSkJut390BqVDfRKg4OAHfgPKyf9sIPkl6ioaxhFCEc4NbCyuccSf5xRUNVAYJB5c98YMumFwqGiBz6H09lQk=', 'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOni4yEAR1WulQ0qK868whKpIbAuHVwVcRlsWDswMtE+hdMc1ichRhUwpUyRfbVf9GAPksxrWMZORAoAu2/LjRM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:51:13.052 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:51:13.075 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:51:13.097 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:51:13.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:51:13.164 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:13.165 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:51:13.213 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:51:13.213 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 10:51:08.060240690 +0000 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 10:48:06.738417382 +0000 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 10:48:06.738417382 +0000 2024-03-29T10:51:13.269 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 10:48:06.738417382 +0000 2024-03-29T10:51:13.270 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:51:13.334 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T10:51:13.334 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T10:51:13.334 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000183044 s, 2.8 MB/s 2024-03-29T10:51:13.335 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:51:13.390 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-03-29T10:51:13.446 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:51:13.446 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:13.446 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T10:51:13.446 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:13.446 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:13.446 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 10:51:08.061240670 +0000 2024-03-29T10:51:13.447 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 10:48:07.082410268 +0000 2024-03-29T10:51:13.447 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 10:48:07.082410268 +0000 2024-03-29T10:51:13.447 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 10:48:07.082410268 +0000 2024-03-29T10:51:13.447 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:51:13.512 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T10:51:13.512 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T10:51:13.512 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000155444 s, 3.3 MB/s 2024-03-29T10:51:13.513 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:51:13.568 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 10:51:08.061240670 +0000 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 10:48:07.441402843 +0000 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 10:48:07.441402843 +0000 2024-03-29T10:51:13.623 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 10:48:07.441402843 +0000 2024-03-29T10:51:13.624 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:51:13.688 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T10:51:13.688 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T10:51:13.688 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000216752 s, 2.4 MB/s 2024-03-29T10:51:13.689 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:51:13.745 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-29 10:51:08.062240650 +0000 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-29 10:48:07.799395440 +0000 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-29 10:48:07.799395440 +0000 2024-03-29T10:51:13.800 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-29 10:48:07.799395440 +0000 2024-03-29T10:51:13.801 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:51:13.866 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-29T10:51:13.866 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-29T10:51:13.866 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000191049 s, 2.7 MB/s 2024-03-29T10:51:13.867 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:51:13.923 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:13.923 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:13.923 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:51:13.942 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:51:13.942 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-29T10:51:14.000 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:51:14.000 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:14.000 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:51:14.000 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:14.000 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:14.000 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 10:50:49.645756831 +0000 2024-03-29T10:51:14.001 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 10:48:14.165056292 +0000 2024-03-29T10:51:14.001 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 10:48:14.165056292 +0000 2024-03-29T10:51:14.001 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 10:48:14.165056292 +0000 2024-03-29T10:51:14.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:51:14.070 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T10:51:14.071 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T10:51:14.071 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000331591 s, 1.5 MB/s 2024-03-29T10:51:14.072 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:51:14.131 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-29T10:51:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:51:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:14.190 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T10:51:14.190 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:14.190 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:14.191 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 10:50:49.645756831 +0000 2024-03-29T10:51:14.191 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 10:48:14.502047109 +0000 2024-03-29T10:51:14.191 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 10:48:14.502047109 +0000 2024-03-29T10:51:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 10:48:14.502047109 +0000 2024-03-29T10:51:14.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:51:14.261 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T10:51:14.261 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T10:51:14.261 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000194877 s, 2.6 MB/s 2024-03-29T10:51:14.263 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:51:14.323 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-29T10:51:14.380 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:51:14.381 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:14.381 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T10:51:14.381 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:14.381 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:14.381 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 10:50:49.646756804 +0000 2024-03-29T10:51:14.381 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 10:48:14.833038088 +0000 2024-03-29T10:51:14.382 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 10:48:14.833038088 +0000 2024-03-29T10:51:14.382 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 10:48:14.833038088 +0000 2024-03-29T10:51:14.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:51:14.448 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T10:51:14.448 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T10:51:14.448 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00016177 s, 3.2 MB/s 2024-03-29T10:51:14.449 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:51:14.507 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-29 10:50:49.646756804 +0000 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-29 10:48:15.172028850 +0000 2024-03-29T10:51:14.562 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-29 10:48:15.172028850 +0000 2024-03-29T10:51:14.563 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-29 10:48:15.172028850 +0000 2024-03-29T10:51:14.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:51:14.629 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-29T10:51:14.630 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-29T10:51:14.630 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000251262 s, 2.0 MB/s 2024-03-29T10:51:14.634 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:51:14.690 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:14.691 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:51:14.691 INFO:tasks.ceph:Generating config... 2024-03-29T10:51:14.692 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:51:14.692 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:51:14.692 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:51:14.692 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:51:14.692 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:51:14.692 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:51:14.692 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:51:14.692 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:51:14.692 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:51:14.693 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:51:14.693 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:51:14.693 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:51:14.693 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:51:14.693 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:51:14.693 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:51:14.693 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:51:14.694 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:51:14.694 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:51:14.694 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:51:14.694 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:51:14.694 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:51:14.694 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:51:14.694 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:51:14.694 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:51:14.694 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:51:14.694 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:51:14.694 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:51:14.695 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:51:14.696 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:51:14.696 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:51:14.696 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:51:14.696 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:51:14.696 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:51:14.696 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:51:14.697 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:51:14.697 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:14.751 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:51:14.754 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:51:14.801 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:51:14.829 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.155:6789]'), ('mon.c', '[v1:172.21.15.155:6790]'), ('mon.b', '[v1:172.21.15.183:6789]')] 2024-03-29T10:51:14.830 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.183:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:51:14.831 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:14.831 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:51:14.885 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.155:6789]' --addv c '[v1:172.21.15.155:6790]' --addv b '[v1:172.21.15.183:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:14.970 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:14.970 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: generated fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:14.970 INFO:teuthology.orchestra.run.smithi155.stdout:setting min_mon_release = quincy 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:epoch 0 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:min_mon_release 17 (quincy) 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:election_strategy: 1 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:14.971 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:51:14.973 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:51:14.987 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7... 2024-03-29T10:51:14.988 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:51:15.029 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:51:15.077 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2024-03-29T10:51:15.077 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2024-03-29T10:51:15.077 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:51:15.078 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2024-03-29T10:51:15.079 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.183:6789]" 2024-03-29T10:51:15.080 INFO:teuthology.orchestra.run.smithi155.stdout: lockdep = True 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: mon client directed command retry = 5 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: mon election default strategy = 1 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: ms bind msgr2 = False 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: ms inject socket failures = 5000 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: ms type = async 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 600 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.081 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug trim objects = true 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2024-03-29T10:51:15.082 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression algorithm = zstd 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression mode = aggressive 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore zero block detection = True 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 20 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 20 2024-03-29T10:51:15.083 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 10 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd class default list = * 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd class load list = * 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd client watch timeout = 120 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify cached snaps = True 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify missing on start = True 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock override recovery settings = True 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2024-03-29T10:51:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue = debug_random 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue cut off = debug_random 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: osd shutdown pgref assert = True 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2024-03-29T10:51:15.085 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.086 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon min osdmap epochs = 50 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune interval = 2 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune min = 15 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon scrub interval = 300 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on pool no app = False 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: paxos service trim min = 10 2024-03-29T10:51:15.087 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: debug objecter = 20 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout: debug rados = 20 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2024-03-29T10:51:15.088 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2024-03-29T10:51:15.089 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-29T10:51:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-29T10:51:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:51:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-29T10:51:15.089 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:51:15.090 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-29T10:51:15.091 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "[v1:172.21.15.155:6789],[v1:172.21.15.155:6790],[v1:172.21.15.183:6789]" 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: lockdep = True 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: mon client directed command retry = 5 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: mon election default strategy = 1 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: ms bind msgr2 = False 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: ms inject socket failures = 5000 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: ms type = async 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-29T10:51:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-29T10:51:15.094 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:51:15.095 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-29T10:51:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-29T10:51:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-29T10:51:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-29T10:51:15.096 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression algorithm = zstd 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression mode = aggressive 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore zero block detection = True 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 20 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 20 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 10 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: osd class default list = * 2024-03-29T10:51:15.097 INFO:teuthology.orchestra.run.smithi183.stdout: osd class load list = * 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd client watch timeout = 120 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify cached snaps = True 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify missing on start = True 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock override recovery settings = True 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue = debug_random 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue cut off = debug_random 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: osd shutdown pgref assert = True 2024-03-29T10:51:15.098 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-29T10:51:15.099 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-29T10:51:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-29T10:51:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-29T10:51:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-29T10:51:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:51:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-29T10:51:15.100 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:51:15.101 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: mon min osdmap epochs = 50 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune interval = 2 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune min = 15 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: mon scrub interval = 300 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on pool no app = False 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: paxos service trim min = 10 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-29T10:51:15.102 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: debug objecter = 20 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout: debug rados = 20 2024-03-29T10:51:15.103 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-29T10:51:15.104 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-29T10:51:15.104 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-29T10:51:15.105 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:51:15.106 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:51:15.162 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:51:15.162 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:15.162 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:51:15.177 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:15.177 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:51:15.232 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T10:51:15.232 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:15.232 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:51:15.232 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:51:15.311 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:15.311 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:15.365 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T10:51:15.366 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:15.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:51:15.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:51:15.408 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:15.408 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:15.465 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:51:15.465 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:51:15.465 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:51:15.520 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:51:15.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:51:15.586 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:51:15.588 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:51:15.588 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:51:15.589 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:51:15.631 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:51:15.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:51:15.689 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:51:15.704 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:51:15.704 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:51:15.704 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:15.731 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:15.732 INFO:tasks.ceph:role: osd.0 2024-03-29T10:51:15.732 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T10:51:15.732 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:51:15.806 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:15.806 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:15.806 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:15.806 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:15.806 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:15.806 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:15.806 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:15.807 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:15.807 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:15.807 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:16.333 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T10:51:16.335 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T10:51:16.335 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:16.469 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:16.503 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:16.572 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:16.572 INFO:tasks.ceph:role: osd.1 2024-03-29T10:51:16.572 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T10:51:16.572 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:51:16.647 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:16.647 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:16.647 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:16.647 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:16.647 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:16.647 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:16.647 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:16.648 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:16.648 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:16.648 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:17.090 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T10:51:17.092 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T10:51:17.092 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:17.182 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:17.214 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:17.282 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:17.282 INFO:tasks.ceph:role: osd.2 2024-03-29T10:51:17.282 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T10:51:17.283 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:51:17.356 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:17.356 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:17.356 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:17.357 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:17.357 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:17.357 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:17.357 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:17.357 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:17.357 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:17.357 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:17.844 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T10:51:17.846 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T10:51:17.846 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:17.980 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:18.014 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:18.081 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:18.081 INFO:tasks.ceph:role: osd.3 2024-03-29T10:51:18.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T10:51:18.081 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:51:18.155 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:18.155 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:18.155 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:18.155 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:18.155 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:18.156 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:18.156 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:18.156 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:18.156 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:18.156 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:18.681 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-03-29T10:51:18.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-03-29T10:51:18.683 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:18.802 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:18.834 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:18.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:18.947+0000 7fc951013640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:51:18.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:18.947+0000 7fc951013640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:51:18.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:18.947+0000 7fc951013640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:51:18.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:18.947+0000 7fc951013640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:51:18.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:18.947+0000 7fc951013640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:51:18.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:18.948+0000 7fc951013640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:51:21.781 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:51:21.812 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:21.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:21.921+0000 7f23e1793640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:51:21.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:21.922+0000 7f23e1793640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:51:21.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:21.922+0000 7f23e1793640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:51:21.905 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:21.922+0000 7f23e1793640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:51:21.905 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:21.922+0000 7f23e1793640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:51:21.905 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:21.922+0000 7f23e1793640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:51:24.758 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:51:24.792 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:24.885 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:24.902+0000 7f413d20f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:51:24.886 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:24.902+0000 7f413d20f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:51:24.886 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:24.902+0000 7f413d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:51:24.886 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:24.902+0000 7f413d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:51:24.886 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:24.902+0000 7f413d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:51:24.886 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:24.903+0000 7f413d20f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:51:27.733 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:51:27.763 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:27.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:27.871+0000 7fb6a820d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:51:27.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:27.871+0000 7fb6a820d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:51:27.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:27.871+0000 7fb6a820d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:51:27.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:27.871+0000 7fb6a820d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:51:27.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:27.871+0000 7fb6a820d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:51:27.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:27.872+0000 7fb6a820d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:51:30.703 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:51:30.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:30.770 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:30.770 INFO:tasks.ceph:role: osd.4 2024-03-29T10:51:30.770 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T10:51:30.770 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:51:30.847 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:30.847 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:30.847 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:30.848 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:30.848 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:30.848 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:30.848 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:30.848 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:30.848 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:30.848 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:31.359 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T10:51:31.362 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T10:51:31.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:31.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:31.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:31.625 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:31.625 INFO:tasks.ceph:role: osd.5 2024-03-29T10:51:31.625 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T10:51:31.625 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:31.701 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:32.152 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T10:51:32.154 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T10:51:32.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:32.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:32.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:32.361 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:32.362 INFO:tasks.ceph:role: osd.6 2024-03-29T10:51:32.362 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T10:51:32.362 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:51:32.437 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:32.437 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:32.437 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:32.437 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:32.438 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:32.438 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:32.438 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:32.438 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:32.438 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:32.438 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:32.933 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T10:51:32.935 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T10:51:32.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:33.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:33.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:33.182 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:51:33.182 INFO:tasks.ceph:role: osd.7 2024-03-29T10:51:33.182 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T10:51:33.183 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:51:33.259 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:51:33.260 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:51:33.769 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-29T10:51:33.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-29T10:51:33.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:33.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:33.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:34.031 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:34.049+0000 7fabc5181640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:51:34.032 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:34.051+0000 7fabc5181640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:51:34.032 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:34.051+0000 7fabc5181640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:51:34.032 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:34.051+0000 7fabc5181640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:51:34.032 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:34.051+0000 7fabc5181640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:51:34.033 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:34.051+0000 7fabc5181640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:51:36.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:51:36.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:37.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:37.027+0000 7fde7799a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:51:37.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:37.027+0000 7fde7799a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:51:37.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:37.028+0000 7fde7799a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:51:37.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:37.028+0000 7fde7799a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:51:37.010 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:37.028+0000 7fde7799a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:51:37.011 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:37.028+0000 7fde7799a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:51:39.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:51:39.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:39.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:40.002+0000 7fbf4162a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:51:39.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:40.003+0000 7fbf4162a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:51:39.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:40.003+0000 7fbf4162a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:51:39.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:40.003+0000 7fbf4162a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:51:39.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:40.003+0000 7fbf4162a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:51:39.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:40.003+0000 7fbf4162a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:51:42.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:51:42.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:42.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:42.979+0000 7f9ba149d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:51:42.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:42.980+0000 7f9ba149d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:51:42.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:42.980+0000 7f9ba149d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:51:42.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:42.980+0000 7f9ba149d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:51:42.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:42.980+0000 7f9ba149d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:51:42.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:42.980+0000 7f9ba149d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:51:45.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:51:45.837 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:51:45.837 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:45.837 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:51:45.869 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:45.869 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:51:45.939 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:45.939 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:51:46.008 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:46.008 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:51:46.076 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:46.076 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:51:46.144 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:46.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:51:46.174 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:46.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:51:46.244 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:46.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:51:46.312 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:46.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:51:46.381 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:46.381 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:51:46.448 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:46.448 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:51:46.464 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:46.464 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:51:46.503 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:51:46.503 DEBUG:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:51:46.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:51:46.550 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2024-03-29T10:51:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAjnQZmC3cqIBAAmkPsB73ZN7XlWf1akK2+hw== 2024-03-29T10:51:46.550 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2024-03-29T10:51:46.550 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAmnQZmkQOAOBAALQI+f3P/vE7yhv7jIWVs4Q== 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQApnQZmU68LNxAAP56Vh77BbQhVY9n8N9L7Ng== 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAsnQZmlNDhNRAA0Oizd8t6VJbjfCuwENuhGg== 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAvnQZmZXoLNBAA5Jq0riBUTRzDpe7vdCSifQ== 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAjnQZmUS0cJBAApdKIzfWqkRxp1faazkYBLg== 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA2nQZmBUsDAxAAIrfByGXq/gOAdIAe63XXOg== 2024-03-29T10:51:46.551 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA5nQZm5YmzARAA6/gRPGHtiMGbe5YBjooSSQ== 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA8nQZm8NI4ABAAOzop57tXTwgJbisTqX5ITQ== 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA+nQZmG6dzOhAAmD+1ZKmwr0H80ORkYUwBlQ== 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAjnQZmnIbUJhAAOfvaaywvC9iYg+qStq+QpA== 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout:[client.1] 2024-03-29T10:51:46.552 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAjnQZm4r5AKhAAlD7/Sg4LBsnU47lRTTlj1A== 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.y] 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAjnQZmC3cqIBAAmkPsB73ZN7XlWf1akK2+hw== 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAmnQZmkQOAOBAALQI+f3P/vE7yhv7jIWVs4Q== 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQApnQZmU68LNxAAP56Vh77BbQhVY9n8N9L7Ng== 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAsnQZmlNDhNRAA0Oizd8t6VJbjfCuwENuhGg== 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAvnQZmZXoLNBAA5Jq0riBUTRzDpe7vdCSifQ== 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAjnQZmUS0cJBAApdKIzfWqkRxp1faazkYBLg== 2024-03-29T10:51:46.569 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.4] 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA2nQZmBUsDAxAAIrfByGXq/gOAdIAe63XXOg== 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.5] 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA5nQZm5YmzARAA6/gRPGHtiMGbe5YBjooSSQ== 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.6] 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA8nQZm8NI4ABAAOzop57tXTwgJbisTqX5ITQ== 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.7] 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQA+nQZmG6dzOhAAmD+1ZKmwr0H80ORkYUwBlQ== 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAjnQZmnIbUJhAAOfvaaywvC9iYg+qStq+QpA== 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-03-29T10:51:46.570 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAjnQZm4r5AKhAAlD7/Sg4LBsnU47lRTTlj1A== 2024-03-29T10:51:46.571 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:46.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:46.664 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.667 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.720 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.775 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.829 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.831 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.882 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:46.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:51:46.936 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.990 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:46.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:47.044 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:47.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:47.097 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:47.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:51:47.151 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:47.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:47.208 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:47.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:51:47.265 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:51:47.265 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:51:47.292 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:47.389 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:51:47.416 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:51:47.483 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:47.582 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:51:47.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:51:47.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:51:47.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:51:47.762 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:47.765 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:51:47.817 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:51:47.817 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:51:47.817 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:51:47.820 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:51:47.820 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:51:47.820 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:51:47.822 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:51:47.822 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:51:47.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:51:47.859 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:51:47.860 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:51:47.860 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:51:47.860 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:51:47.862 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:51:47.862 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:51:47.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:51:47.864 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:51:47.865 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:51:47.865 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.986+0000 7f9780855640 10 monclient: get_monmap_and_config 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.986+0000 7f9780855640 10 monclient: build_initial_monmap 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.986+0000 7f9780855640 10 monclient: monmap: 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:47.986785+0000 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:47.986785+0000 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:47.967 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:47.968 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:47.968 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:47.968 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:47.968 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:47.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.986+0000 7f9780855640 1 Processor -- start 2024-03-29T10:51:47.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.986+0000 7f9780855640 1 -- start start 2024-03-29T10:51:47.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.986+0000 7f9780855640 10 monclient: init 2024-03-29T10:51:47.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 10 monclient(hunting): picked mon.noname-c con 0x7f977c0b8240 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 10 monclient(hunting): picked mon.noname-b con 0x7f977c0b1f60 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 10 monclient(hunting): picked mon.noname-a con 0x7f977c09c8c0 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c09bd80 con 0x7f977c09c8c0 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c09b960 con 0x7f977c0b8240 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0b2ec0 con 0x7f977c0b1f60 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9780855640 10 monclient(hunting): authenticate will time out at 811.279477s 2024-03-29T10:51:47.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f977ad9d640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f977c09c8c0 0x7f977c14d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:41656/0 (socket says 172.21.15.155:41656) 2024-03-29T10:51:47.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f977ad9d640 1 -- 172.21.15.155:0/1327616313 learned_addr learned my addr 172.21.15.155:0/1327616313 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:51:47.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:47.987+0000 7f9773fff640 1 --1- 172.21.15.155:0/1327616313 >> v1:172.21.15.155:6790/0 conn(0x7f977c0b1f60 0x7f977c09c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39212/0 (socket says 172.21.15.155:39212) 2024-03-29T10:51:50.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient(hunting): tick 2024-03-29T10:51:50.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:51:50.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 1 monclient(hunting): continuing hunt 2024-03-29T10:51:50.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:51:50.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 1 -- 172.21.15.155:0/1327616313 >> v1:172.21.15.155:6790/0 conn(0x7f977c0b1f60 legacy=0x7f977c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:50.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 1 -- 172.21.15.155:0/1327616313 >> v1:172.21.15.183:6789/0 conn(0x7f977c0b8240 legacy=0x7f977c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 1 -- 172.21.15.155:0/1327616313 >> v1:172.21.15.155:6789/0 conn(0x7f977c09c8c0 legacy=0x7f977c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient(hunting): picked mon.noname-a con 0x7f975c0036e0 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient(hunting): picked mon.noname-b con 0x7f975c0072b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient(hunting): picked mon.noname-c con 0x7f977c0b8240 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 1 -- 172.21.15.155:0/1327616313 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c13fb60 con 0x7f975c0036e0 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 1 -- 172.21.15.155:0/1327616313 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c00f760 con 0x7f977c0b8240 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 1 -- 172.21.15.155:0/1327616313 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0109c0 con 0x7f975c0072b0 2024-03-29T10:51:50.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:50.987+0000 7f9778d99640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.157+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755705352 0 0) 0x7f977c13fb60 con 0x7f975c0036e0 2024-03-29T10:51:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9768003520 con 0x7f975c0036e0 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910001538 0 0) 0x7f975c0109c0 con 0x7f975c0072b0 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f977c13fb60 con 0x7f975c0072b0 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1146067128 0 0) 0x7f977c13fb60 con 0x7f975c0072b0 2024-03-29T10:51:53.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 >> v1:172.21.15.183:6789/0 conn(0x7f977c0b8240 legacy=0x7f975c00cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 >> v1:172.21.15.155:6789/0 conn(0x7f975c0036e0 legacy=0x7f975c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 monclient: found mon.noname-b 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c011c20 con 0x7f975c0072b0 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: _finish_auth 0 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9764001610 con 0x7f975c0072b0 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: dump: 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9780855640 5 monclient: authenticate success, global_id 4106 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9780855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9764001770 con 0x7f975c0072b0 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.158+0000 7f9779d9b640 1 -- 172.21.15.155:0/1327616313 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9764001b80 con 0x7f975c0072b0 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.159+0000 7f9780855640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.159+0000 7f9780855640 4 set_mon_vals no callback set 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.159+0000 7f9779d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.159+0000 7f9779d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.159+0000 7f9779d9b640 10 monclient: dump: 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:53.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.160+0000 7f9780855640 10 monclient: shutdown 2024-03-29T10:51:53.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.160+0000 7f9780855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.160+0000 7f9780855640 1 -- 172.21.15.155:0/1327616313 >> v1:172.21.15.155:6790/0 conn(0x7f975c0072b0 legacy=0x7f975c0096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.160+0000 7f9780855640 1 -- 172.21.15.155:0/1327616313 shutdown_connections 2024-03-29T10:51:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.160+0000 7f9780855640 1 -- 172.21.15.155:0/1327616313 >> 172.21.15.155:0/1327616313 conn(0x7f977c066b00 msgr2=0x7f977c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.160+0000 7f9780855640 1 -- 172.21.15.155:0/1327616313 shutdown_connections 2024-03-29T10:51:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.160+0000 7f9780855640 1 -- 172.21.15.155:0/1327616313 wait complete. 2024-03-29T10:51:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.161+0000 7f9780855640 10 monclient: build_initial_monmap 2024-03-29T10:51:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.161+0000 7f9780855640 10 monclient: monmap: 2024-03-29T10:51:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T10:51:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-03-29T10:51:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-03-29T10:51:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.161+0000 7f9780855640 1 librados: starting msgr at 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.161+0000 7f9780855640 1 librados: starting objecter 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.161+0000 7f9780855640 1 Processor -- start 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.161+0000 7f9780855640 1 -- start start 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 1 librados: setting wanted keys 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 1 librados: calling monclient init 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient: init 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient(hunting): picked mon.noname-a con 0x7f975c0036e0 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient(hunting): picked mon.noname-b con 0x7f975c0072b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient(hunting): picked mon.noname-c con 0x7f977c0b8240 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c140440 con 0x7f975c0036e0 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c139af0 con 0x7f975c0072b0 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c139c80 con 0x7f977c0b8240 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9780855640 10 monclient(hunting): authenticate will time out at 816.454648s 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f977a59c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f975c0036e0 0x7f977c140960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49976/0 (socket says 172.21.15.155:49976) 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f977a59c640 1 -- 172.21.15.155:0/2486185789 learned_addr learned my addr 172.21.15.155:0/2486185789 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.162+0000 7f9773fff640 1 -- 172.21.15.155:0/2486185789 >> v1:172.21.15.183:6789/0 conn(0x7f975c0072b0 legacy=0x7f977c141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:51:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9773fff640 1 -- 172.21.15.155:0/2486185789 >> v1:172.21.15.183:6789/0 conn(0x7f975c0072b0 legacy=0x7f977c141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9773fff640 1 --1- 172.21.15.155:0/2486185789 >> v1:172.21.15.183:6789/0 conn(0x7f975c0072b0 0x7f977c141070 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9773fff640 1 --1- 172.21.15.155:0/2486185789 >> v1:172.21.15.183:6789/0 conn(0x7f975c0072b0 0x7f977c141070 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9772ffd640 10 client.?.objecter ms_handle_connect 0x7f975c0036e0 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9772ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9772ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.183:6789/0 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9772ffd640 10 client.?.objecter ms_handle_connect 0x7f977c0b8240 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.163+0000 7f9772ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320265710 0 0) 0x7f977c139c80 con 0x7f977c0b8240 2024-03-29T10:51:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c0101c0 con 0x7f977c0b8240 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1989008984 0 0) 0x7f975c0101c0 con 0x7f977c0b8240 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f977c139c80 con 0x7f977c0b8240 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9774001ba0 con 0x7f977c0b8240 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847254042 0 0) 0x7f977c140440 con 0x7f975c0036e0 2024-03-29T10:51:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.164+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c0101c0 con 0x7f975c0036e0 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3166171471 0 0) 0x7f975c0101c0 con 0x7f975c0036e0 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f977c140440 con 0x7f975c0036e0 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f976c001970 con 0x7f975c0036e0 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:53.160 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 763160770 0 0) 0x7f977c140440 con 0x7f975c0036e0 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 >> v1:172.21.15.155:6790/0 conn(0x7f977c0b8240 legacy=0x7f977c1e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 >> v1:172.21.15.183:6789/0 conn(0x7f975c0072b0 legacy=0x7f977c141070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 monclient: found mon.a 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c139e10 con 0x7f975c0036e0 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:51:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 10 monclient: _renew_subs 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c142c30 con 0x7f975c0036e0 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 10 monclient: _renew_subs 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c13fe20 con 0x7f975c0036e0 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9780855640 1 librados: init done 2024-03-29T10:51:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f976c0035f0 con 0x7f975c0036e0 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f9772ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.165+0000 7f977959a640 4 set_mon_vals no callback set 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.166+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f976c003a00 con 0x7f975c0036e0 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.166+0000 7f9772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.166+0000 7f9772ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.166+0000 7f9772ffd640 10 monclient: dump: 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.164 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9780855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9780855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9780855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c09b170 con 0x7f975c0036e0 2024-03-29T10:51:53.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f976c003ef0 con 0x7f975c0036e0 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 260173879 0 0) 0x7f976c006410 con 0x7f975c0036e0 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9772ffd640 10 client.4113.objecter ms_dispatch 0x7f977c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9772ffd640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9772ffd640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.167+0000 7f9772ffd640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.170+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f976c0041a0 con 0x7f975c0036e0 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.170+0000 7f9772ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.170+0000 7f9772ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:53.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.243+0000 7f9780855640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:51:53.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.243+0000 7f9780855640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:51:53.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.243+0000 7f9780855640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:51:53.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.243+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f977c04f730 con 0x7f975c0036e0 2024-03-29T10:51:53.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.245+0000 7f9772ffd640 1 -- 172.21.15.155:0/2486185789 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f976c004410 con 0x7f975c0036e0 2024-03-29T10:51:53.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.245+0000 7f9772ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:51:53.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.245+0000 7f9772ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:51:53.226 INFO:teuthology.orchestra.run.smithi155.stderr:adjusted tunables profile to default 2024-03-29T10:51:53.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.245+0000 7f9780855640 10 librados: watch_flush enter 2024-03-29T10:51:53.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 10 librados: watch_flush exit 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 10 monclient: shutdown 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 >> v1:172.21.15.155:6789/0 conn(0x7f975c0036e0 legacy=0x7f977c140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 shutdown_connections 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 >> 172.21.15.155:0/2486185789 conn(0x7f977c066b00 msgr2=0x7f977c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 shutdown_connections 2024-03-29T10:51:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 1 -- 172.21.15.155:0/2486185789 wait complete. 2024-03-29T10:51:53.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:53.246+0000 7f9780855640 1 librados: shutdown 2024-03-29T10:51:53.238 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:51:53.239 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:51:53.239 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:53.239 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:51:53.267 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:53.267 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:51:53.334 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:53.334 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:51:53.401 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-29T10:51:53.401 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:51:53.468 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:53.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:51:53.497 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:53.498 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:51:53.572 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:53.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:51:53.614 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-29T10:51:53.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:51:53.622 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:53.640+0000 7f5afda2d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:51:53.653 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new f08abf9e-cefc-4532-9ddd-4b175fc54b03 0 2024-03-29T10:51:53.666 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:53.685+0000 7f4900421040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:51:53.678 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:53.697+0000 7f5afda2d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:51:53.720 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:53.738+0000 7f4900421040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.796+0000 7fd8e2ca5640 10 monclient: get_monmap_and_config 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.796+0000 7fd8e2ca5640 10 monclient: build_initial_monmap 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.796+0000 7fd8e2ca5640 10 monclient: monmap: 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:53.797569+0000 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:53.797569+0000 2024-03-29T10:51:53.778 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:53.779 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.779 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:53.779 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:53.779 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:53.779 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.797+0000 7fd8e2ca5640 1 Processor -- start 2024-03-29T10:51:53.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.797+0000 7fd8e2ca5640 1 -- start start 2024-03-29T10:51:53.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.797+0000 7fd8e2ca5640 10 monclient: init 2024-03-29T10:51:53.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.797+0000 7fd8e2ca5640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:53.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.797+0000 7fd8e2ca5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fd8dc09ac40 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fd8dc09c870 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fd8dc0b4360 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc09dec0 con 0x7fd8dc09c870 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc09da30 con 0x7fd8dc09ac40 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc0b39c0 con 0x7fd8dc0b4360 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 10 monclient(hunting): authenticate will time out at 831.045996s 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e0a41640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd8dc09ac40 0x7fd8dc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34692/0 (socket says 172.21.15.183:34692) 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e0a41640 1 -- 172.21.15.183:0/3711806569 learned_addr learned my addr 172.21.15.183:0/3711806569 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837912881 0 0) 0x7fd8dc09da30 con 0x7fd8dc09ac40 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): my global_id is 4102 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8cc003520 con 0x7fd8dc09ac40 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429332947 0 0) 0x7fd8c8001110 con 0x7fd8dc09c870 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8dc09da30 con 0x7fd8dc09c870 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392476131 0 0) 0x7fd8cc003520 con 0x7fd8dc09ac40 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 >> v1:172.21.15.155:6790/0 conn(0x7fd8dc0b4360 legacy=0x7fd8dc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 >> v1:172.21.15.155:6789/0 conn(0x7fd8dc09c870 legacy=0x7fd8dc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 monclient: found mon.noname-c 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:53.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0b5160 con 0x7fd8dc09ac40 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 5 monclient: authenticate success, global_id 4102 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd8d0001690 con 0x7fd8dc09ac40 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: dump: 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.790 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8d00017f0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 1 -- 172.21.15.183:0/3711806569 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd8d0001c40 con 0x7fd8dc09ac40 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8e2ca5640 4 set_mon_vals no callback set 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.798+0000 7fd8db7fe640 10 monclient: dump: 2024-03-29T10:51:53.791 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 10 monclient: shutdown 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3711806569 >> v1:172.21.15.183:6789/0 conn(0x7fd8dc09ac40 legacy=0x7fd8dc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3711806569 shutdown_connections 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3711806569 >> 172.21.15.183:0/3711806569 conn(0x7fd8dc066b00 msgr2=0x7fd8dc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:53.792 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3711806569 shutdown_connections 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3711806569 wait complete. 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 10 monclient: build_initial_monmap 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.799+0000 7fd8e2ca5640 10 monclient: monmap: 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:53.793 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 librados: starting msgr at 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 librados: starting objecter 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 Processor -- start 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 -- start start 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 librados: setting wanted keys 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 librados: calling monclient init 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient: init 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fd8dc09ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fd8dc137600 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fd8dc1dafc0 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc1401c0 con 0x7fd8dc137600 2024-03-29T10:51:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc140600 con 0x7fd8dc1dafc0 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8dc1408e0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.800+0000 7fd8e2ca5640 10 monclient(hunting): authenticate will time out at 831.048909s 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e1242640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd8dc1dafc0 0x7fd8dc140e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34698/0 (socket says 172.21.15.183:34698) 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e1242640 1 -- 172.21.15.183:0/3017645484 learned_addr learned my addr 172.21.15.183:0/3017645484 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 client.?.objecter ms_handle_connect 0x7fd8dc1dafc0 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 client.?.objecter ms_handle_connect 0x7fd8dc09ac40 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 client.?.objecter ms_handle_connect 0x7fd8dc137600 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882246136 0 0) 0x7fd8dc140600 con 0x7fd8dc1dafc0 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.795 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c0004eb0 con 0x7fd8dc1dafc0 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191304585 0 0) 0x7fd8dc1408e0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8dc140600 con 0x7fd8dc09ac40 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119715760 0 0) 0x7fd8dc1401c0 con 0x7fd8dc137600 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8dc1408e0 con 0x7fd8dc137600 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1152981498 0 0) 0x7fd8dc140600 con 0x7fd8dc09ac40 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8dc1401c0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 796871857 0 0) 0x7fd8c0004eb0 con 0x7fd8dc1dafc0 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8dc140600 con 0x7fd8dc1dafc0 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd8d0001b00 con 0x7fd8dc09ac40 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): dump: 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 765454250 0 0) 0x7fd8dc1408e0 con 0x7fd8dc137600 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8c0004eb0 con 0x7fd8dc137600 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd8d4001c70 con 0x7fd8dc1dafc0 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): dump: 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.798 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd8c80018a0 con 0x7fd8dc137600 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient(hunting): dump: 2024-03-29T10:51:53.799 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.800 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 639307851 0 0) 0x7fd8dc1401c0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 >> v1:172.21.15.183:6789/0 conn(0x7fd8dc1dafc0 legacy=0x7fd8dc140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 >> v1:172.21.15.155:6789/0 conn(0x7fd8dc137600 legacy=0x7fd8dc1da8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 monclient: found mon.c 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc1db410 con 0x7fd8dc09ac40 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: _finish_auth 0 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 5 monclient: authenticate success, global_id 4118 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 10 monclient: _renew_subs 2024-03-29T10:51:53.801 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc141510 con 0x7fd8dc09ac40 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 10 monclient: _renew_subs 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc141a90 con 0x7fd8dc09ac40 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8e2ca5640 1 librados: init done 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8d0004a90 con 0x7fd8dc09ac40 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd8d0004da0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8daffd640 4 set_mon_vals no callback set 2024-03-29T10:51:53.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 monclient: dump: 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd8d0005290 con 0x7fd8dc09ac40 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1847965601 0 0) 0x7fd8d0006670 con 0x7fd8dc09ac40 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.801+0000 7fd8d97fa640 10 client.4118.objecter ms_dispatch 0x7fd8dc000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:51:53.803 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.802+0000 7fd8d97fa640 3 client.4118.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.802+0000 7fd8d97fa640 3 client.4118.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.802+0000 7fd8d97fa640 20 client.4118.objecter dump_active .. 0 homeless 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.802+0000 7fd8e2ca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.802+0000 7fd8e2ca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.802+0000 7fd8e2ca5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.802+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc005080 con 0x7fd8dc09ac40 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.804+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd8d0006950 con 0x7fd8dc09ac40 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.804+0000 7fd8d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:53.804 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.804+0000 7fd8d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:53.857 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.875+0000 7fd8e2ca5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f08abf9e-cefc-4532-9ddd-4b175fc54b03", "id": 0}] 2024-03-29T10:51:53.857 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.875+0000 7fd8e2ca5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f08abf9e-cefc-4532-9ddd-4b175fc54b03", "id": 0}] 2024-03-29T10:51:53.857 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.875+0000 7fd8e2ca5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:53.857 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.875+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f08abf9e-cefc-4532-9ddd-4b175fc54b03", "id": 0} v 0) v1 -- 0x7fd8bc0058b0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.878+0000 7fd8d97fa640 1 -- 172.21.15.183:0/3017645484 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f08abf9e-cefc-4532-9ddd-4b175fc54b03", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1680105240 0 732707515) 0x7fd8d0006af0 con 0x7fd8dc09ac40 2024-03-29T10:51:53.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.878+0000 7fd8d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f08abf9e-cefc-4532-9ddd-4b175fc54b03", "id": 0}] 2024-03-29T10:51:53.860 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.878+0000 7fd8d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:53.860 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-03-29T10:51:53.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 10 librados: watch_flush enter 2024-03-29T10:51:53.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 10 librados: watch_flush exit 2024-03-29T10:51:53.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 20 client.4118.objecter shutdown clearing up homeless session... 2024-03-29T10:51:53.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 10 client.4118.objecter successfully canceled tick 2024-03-29T10:51:53.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 10 monclient: shutdown 2024-03-29T10:51:53.861 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:53.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 >> v1:172.21.15.155:6790/0 conn(0x7fd8dc09ac40 legacy=0x7fd8dc136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 shutdown_connections 2024-03-29T10:51:53.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 >> 172.21.15.183:0/3017645484 conn(0x7fd8dc066b00 msgr2=0x7fd8dc13e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:53.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 shutdown_connections 2024-03-29T10:51:53.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.879+0000 7fd8e2ca5640 1 -- 172.21.15.183:0/3017645484 wait complete. 2024-03-29T10:51:53.862 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.880+0000 7fd8e2ca5640 1 librados: shutdown 2024-03-29T10:51:53.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new b673f42b-b29e-4994-bc55-f4ed0a62e5c7 1 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.990+0000 7f120f8fb640 10 monclient: get_monmap_and_config 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.990+0000 7f120f8fb640 10 monclient: build_initial_monmap 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.990+0000 7f120f8fb640 10 monclient: monmap: 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:53.991421+0000 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:53.991421+0000 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.989 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 1 Processor -- start 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 1 -- start start 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient: init 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient(hunting): picked mon.noname-b con 0x7f120809ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient(hunting): picked mon.noname-a con 0x7f120809c870 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient(hunting): picked mon.noname-c con 0x7f12080b4360 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120809dec0 con 0x7f120809c870 2024-03-29T10:51:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120809da30 con 0x7f12080b4360 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12080b39c0 con 0x7f120809ac40 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.991+0000 7f120f8fb640 10 monclient(hunting): authenticate will time out at 831.239896s 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f120de98640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f12080b4360 0x7f120814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34710/0 (socket says 172.21.15.183:34710) 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f120de98640 1 -- 172.21.15.183:0/8314212 learned_addr learned my addr 172.21.15.183:0/8314212 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302075234 0 0) 0x7f120809da30 con 0x7f12080b4360 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f4003520 con 0x7f12080b4360 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197489102 0 0) 0x7f1204001110 con 0x7f120809c870 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.991 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f120809da30 con 0x7f120809c870 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431423075 0 0) 0x7f12080b39c0 con 0x7f120809ac40 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:51:53.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1204001110 con 0x7f120809ac40 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2694758425 0 0) 0x7f11f4003520 con 0x7f12080b4360 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 >> v1:172.21.15.155:6790/0 conn(0x7f120809ac40 legacy=0x7f12080b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 >> v1:172.21.15.155:6789/0 conn(0x7f120809c870 legacy=0x7f120813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 monclient: found mon.noname-c 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080b5160 con 0x7f12080b4360 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: _finish_auth 0 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1200001a10 con 0x7f12080b4360 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f120f8fb640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:51:53.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f120f8fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: dump: 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f120f8fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1200001bf0 con 0x7f12080b4360 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f120f8fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 1 -- 172.21.15.183:0/8314212 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1200002000 con 0x7f12080b4360 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f120f8fb640 4 set_mon_vals no callback set 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.992+0000 7f11fffff640 10 monclient: dump: 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.995 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 10 monclient: shutdown 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 1 -- 172.21.15.183:0/8314212 >> v1:172.21.15.183:6789/0 conn(0x7f12080b4360 legacy=0x7f120814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 1 -- 172.21.15.183:0/8314212 shutdown_connections 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 1 -- 172.21.15.183:0/8314212 >> 172.21.15.183:0/8314212 conn(0x7f1208066b00 msgr2=0x7f1208066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 1 -- 172.21.15.183:0/8314212 shutdown_connections 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 1 -- 172.21.15.183:0/8314212 wait complete. 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 10 monclient: build_initial_monmap 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.993+0000 7f120f8fb640 10 monclient: monmap: 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 librados: starting msgr at 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 librados: starting objecter 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 Processor -- start 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 -- start start 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 librados: setting wanted keys 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 librados: calling monclient init 2024-03-29T10:51:53.997 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient: init 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient(hunting): picked mon.noname-c con 0x7f120809ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient(hunting): picked mon.noname-b con 0x7f12081417d0 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1208139650 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1208143160 con 0x7f1208139650 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1208141e80 con 0x7f12081417d0 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1208142310 con 0x7f120809ac40 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120f8fb640 10 monclient(hunting): authenticate will time out at 831.242718s 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120ce96640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f12081417d0 0x7f1208138f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34724/0 (socket says 172.21.15.183:34724) 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f120ce96640 1 -- 172.21.15.183:0/174917895 learned_addr learned my addr 172.21.15.183:0/174917895 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f11fdffb640 10 client.?.objecter ms_handle_connect 0x7f12081417d0 2024-03-29T10:51:53.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.994+0000 7f11fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 client.?.objecter ms_handle_connect 0x7f1208139650 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 client.?.objecter ms_handle_connect 0x7f120809ac40 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828535402 0 0) 0x7f1208141e80 con 0x7f12081417d0 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11ec004eb0 con 0x7f12081417d0 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691809438 0 0) 0x7f1208143160 con 0x7f1208139650 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:53.999 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1208141e80 con 0x7f1208139650 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392665840 0 0) 0x7f1208142310 con 0x7f120809ac40 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1208143160 con 0x7f120809ac40 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218100679 0 0) 0x7f11ec004eb0 con 0x7f12081417d0 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1208142310 con 0x7f12081417d0 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1204001a20 con 0x7f12081417d0 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): dump: 2024-03-29T10:51:54.000 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275955483 0 0) 0x7f1208141e80 con 0x7f1208139650 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11ec004eb0 con 0x7f1208139650 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272183496 0 0) 0x7f1208143160 con 0x7f120809ac40 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1208141e80 con 0x7f120809ac40 2024-03-29T10:51:54.001 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1200001f00 con 0x7f1208139650 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): dump: 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f11f80017f0 con 0x7f120809ac40 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient(hunting): dump: 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.003 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1355781405 0 0) 0x7f1208142310 con 0x7f12081417d0 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 >> v1:172.21.15.155:6790/0 conn(0x7f120809ac40 legacy=0x7f1208140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 >> v1:172.21.15.155:6789/0 conn(0x7f1208139650 legacy=0x7f12080b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 monclient: found mon.b 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1208143390 con 0x7f12081417d0 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 10 monclient: _renew_subs 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12080b0710 con 0x7f12081417d0 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1204001ce0 con 0x7f12081417d0 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1204003a60 con 0x7f12081417d0 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 10 monclient: dump: 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.005 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1204004010 con 0x7f12081417d0 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f11ff7fe640 4 set_mon_vals no callback set 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 10 monclient: _renew_subs 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12081e3c60 con 0x7f12081417d0 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.995+0000 7f120f8fb640 1 librados: init done 2024-03-29T10:51:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1257121984 0 0) 0x7f1204005b70 con 0x7f12081417d0 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f11fdffb640 10 client.4111.objecter ms_dispatch 0x7f1208000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f11fdffb640 3 client.4111.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f11fdffb640 3 client.4111.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f11fdffb640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f120f8fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f120f8fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f120f8fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.996+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120809b020 con 0x7f12081417d0 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.998+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1204005db0 con 0x7f12081417d0 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.998+0000 7f11fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:53.998+0000 7f11fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:54.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.069+0000 7f120f8fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b673f42b-b29e-4994-bc55-f4ed0a62e5c7", "id": 1}] 2024-03-29T10:51:54.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.069+0000 7f120f8fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b673f42b-b29e-4994-bc55-f4ed0a62e5c7", "id": 1}] 2024-03-29T10:51:54.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.069+0000 7f120f8fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.069+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b673f42b-b29e-4994-bc55-f4ed0a62e5c7", "id": 1} v 0) v1 -- 0x7f1208139a50 con 0x7f12081417d0 2024-03-29T10:51:54.055 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.073+0000 7f11fdffb640 1 -- 172.21.15.183:0/174917895 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b673f42b-b29e-4994-bc55-f4ed0a62e5c7", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 393335232 0 940483276) 0x7f1204006440 con 0x7f12081417d0 2024-03-29T10:51:54.055 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.073+0000 7f11fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b673f42b-b29e-4994-bc55-f4ed0a62e5c7", "id": 1}] 2024-03-29T10:51:54.055 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.073+0000 7f11fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:54.055 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-29T10:51:54.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 10 librados: watch_flush enter 2024-03-29T10:51:54.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 10 librados: watch_flush exit 2024-03-29T10:51:54.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 10 monclient: shutdown 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 >> v1:172.21.15.183:6789/0 conn(0x7f12081417d0 legacy=0x7f1208138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 shutdown_connections 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 >> 172.21.15.183:0/174917895 conn(0x7f1208066b00 msgr2=0x7f120813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 shutdown_connections 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 1 -- 172.21.15.183:0/174917895 wait complete. 2024-03-29T10:51:54.057 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.074+0000 7f120f8fb640 1 librados: shutdown 2024-03-29T10:51:54.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new dbb3193a-5420-4a34-8814-4a79c888bd62 2 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.187+0000 7f34863c8640 10 monclient: get_monmap_and_config 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.187+0000 7f34863c8640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.187+0000 7f34863c8640 10 monclient: monmap: 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:54.188449+0000 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:54.188449+0000 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.169 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.170 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:54.170 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:54.170 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.179 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 1 Processor -- start 2024-03-29T10:51:54.179 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 1 -- start start 2024-03-29T10:51:54.179 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient: init 2024-03-29T10:51:54.179 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.179 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:54.179 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient(hunting): picked mon.noname-a con 0x7f3480066b90 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient(hunting): picked mon.noname-c con 0x7f348013f340 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient(hunting): picked mon.noname-b con 0x7f348013f7a0 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348013fc30 con 0x7f3480066b90 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34800afd70 con 0x7f348013f340 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34800aec00 con 0x7f348013f7a0 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.188+0000 7f34863c8640 10 monclient(hunting): authenticate will time out at 831.436898s 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347f7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f348013f340 0x7f3480142060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34728/0 (socket says 172.21.15.183:34728) 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347f7fe640 1 -- 172.21.15.183:0/4217941809 learned_addr learned my addr 172.21.15.183:0/4217941809 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028303642 0 0) 0x7f34800afd70 con 0x7f348013f340 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3468003520 con 0x7f348013f340 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604144125 0 0) 0x7f348013fc30 con 0x7f3480066b90 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34800afd70 con 0x7f3480066b90 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128264633 0 0) 0x7f34800aec00 con 0x7f348013f7a0 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f348013fc30 con 0x7f348013f7a0 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 215513693 0 0) 0x7f3468003520 con 0x7f348013f340 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 >> v1:172.21.15.155:6790/0 conn(0x7f348013f7a0 legacy=0x7f3480142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.182 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 >> v1:172.21.15.155:6789/0 conn(0x7f3480066b90 legacy=0x7f34800af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 monclient: found mon.noname-c 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34800aed90 con 0x7f348013f340 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f34863c8640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:51:54.183 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f34863c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3470001690 con 0x7f348013f340 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: dump: 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.184 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f34863c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3470001870 con 0x7f348013f340 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f34863c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 1 -- 172.21.15.183:0/4217941809 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3470001c80 con 0x7f348013f340 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f34863c8640 4 set_mon_vals no callback set 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.189+0000 7f347effd640 10 monclient: dump: 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.186 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 10 monclient: shutdown 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 1 -- 172.21.15.183:0/4217941809 >> v1:172.21.15.183:6789/0 conn(0x7f348013f340 legacy=0x7f3480142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 1 -- 172.21.15.183:0/4217941809 shutdown_connections 2024-03-29T10:51:54.187 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 1 -- 172.21.15.183:0/4217941809 >> 172.21.15.183:0/4217941809 conn(0x7f34800b4970 msgr2=0x7f34800b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 1 -- 172.21.15.183:0/4217941809 shutdown_connections 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 1 -- 172.21.15.183:0/4217941809 wait complete. 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.190+0000 7f34863c8640 10 monclient: monmap: 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:54.188 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 1 librados: starting msgr at 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 1 librados: starting objecter 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 1 Processor -- start 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 1 -- start start 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 1 librados: setting wanted keys 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 1 librados: calling monclient init 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 10 monclient: init 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.191+0000 7f34863c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:54.189 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient(hunting): picked mon.noname-b con 0x7f3480066b90 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient(hunting): picked mon.noname-a con 0x7f348013f340 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient(hunting): picked mon.noname-c con 0x7f348013f7a0 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3480132a30 con 0x7f348013f340 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34801314b0 con 0x7f3480066b90 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34801d7210 con 0x7f348013f7a0 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient(hunting): authenticate will time out at 831.440023s 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3480066b90 0x7f3480130da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34738/0 (socket says 172.21.15.183:34738) 2024-03-29T10:51:54.190 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347ffff640 1 -- 172.21.15.183:0/1335609681 learned_addr learned my addr 172.21.15.183:0/1335609681 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.191 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 client.?.objecter ms_handle_connect 0x7f3480066b90 2024-03-29T10:51:54.191 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.191 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 client.?.objecter ms_handle_connect 0x7f348013f7a0 2024-03-29T10:51:54.191 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.191 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 client.?.objecter ms_handle_connect 0x7f348013f340 2024-03-29T10:51:54.191 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.191 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22813665 0 0) 0x7f34801314b0 con 0x7f3480066b90 2024-03-29T10:51:54.192 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.192 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.192 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.192 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:51:54.192 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c004eb0 con 0x7f3480066b90 2024-03-29T10:51:54.192 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887700424 0 0) 0x7f3480132a30 con 0x7f348013f340 2024-03-29T10:51:54.192 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.193 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.193 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.193 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:51:54.193 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34801314b0 con 0x7f348013f340 2024-03-29T10:51:54.193 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652512268 0 0) 0x7f34801d7210 con 0x7f348013f7a0 2024-03-29T10:51:54.193 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.193 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480132a30 con 0x7f348013f7a0 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404305913 0 0) 0x7f346c004eb0 con 0x7f3480066b90 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34801d7210 con 0x7f3480066b90 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3474001900 con 0x7f3480066b90 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): dump: 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.194 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2899852763 0 0) 0x7f34801314b0 con 0x7f348013f340 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f346c004eb0 con 0x7f348013f340 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3470001b80 con 0x7f348013f340 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.195 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): dump: 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153166060 0 0) 0x7f3480132a30 con 0x7f348013f7a0 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34801314b0 con 0x7f348013f7a0 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3478001ba0 con 0x7f348013f7a0 2024-03-29T10:51:54.196 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient(hunting): dump: 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1196134017 0 0) 0x7f34801d7210 con 0x7f3480066b90 2024-03-29T10:51:54.197 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 >> v1:172.21.15.155:6790/0 conn(0x7f348013f7a0 legacy=0x7f34801d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 >> v1:172.21.15.155:6789/0 conn(0x7f348013f340 legacy=0x7f34801d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 monclient: found mon.b 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34801d7e40 con 0x7f3480066b90 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f347cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient: _renew_subs 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.192+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34801d3ce0 con 0x7f3480066b90 2024-03-29T10:51:54.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 10 monclient: _renew_subs 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3474003610 con 0x7f3480066b90 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34801d4c50 con 0x7f3480066b90 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 1 librados: init done 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f34740039e0 con 0x7f3480066b90 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 10 monclient: dump: 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.199 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347e7fc640 4 set_mon_vals no callback set 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3474003f50 con 0x7f3480066b90 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1667958515 0 0) 0x7f3474006580 con 0x7f3480066b90 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 10 client.4117.objecter ms_dispatch 0x7f3480000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 3 client.4117.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 3 client.4117.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:51:54.200 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f347cff9640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:51:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.193+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348004f730 con 0x7f3480066b90 2024-03-29T10:51:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.195+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3474004390 con 0x7f3480066b90 2024-03-29T10:51:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.195+0000 7f347cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.195+0000 7f347cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:54.234 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.253+0000 7f5afda2d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:51:54.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.267+0000 7f34863c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dbb3193a-5420-4a34-8814-4a79c888bd62", "id": 2}] 2024-03-29T10:51:54.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.267+0000 7f34863c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dbb3193a-5420-4a34-8814-4a79c888bd62", "id": 2}] 2024-03-29T10:51:54.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.267+0000 7f34863c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.248 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.267+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dbb3193a-5420-4a34-8814-4a79c888bd62", "id": 2} v 0) v1 -- 0x7f348006bb70 con 0x7f3480066b90 2024-03-29T10:51:54.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.271+0000 7f347cff9640 1 -- 172.21.15.183:0/1335609681 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dbb3193a-5420-4a34-8814-4a79c888bd62", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4137862095 0 216599125) 0x7f3474006160 con 0x7f3480066b90 2024-03-29T10:51:54.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.271+0000 7f347cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dbb3193a-5420-4a34-8814-4a79c888bd62", "id": 2}] 2024-03-29T10:51:54.252 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.271+0000 7f347cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:54.253 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-03-29T10:51:54.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.271+0000 7f34863c8640 10 librados: watch_flush enter 2024-03-29T10:51:54.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.271+0000 7f34863c8640 10 librados: watch_flush exit 2024-03-29T10:51:54.254 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.271+0000 7f34863c8640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.271+0000 7f34863c8640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 10 monclient: shutdown 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 >> v1:172.21.15.183:6789/0 conn(0x7f3480066b90 legacy=0x7f3480130da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 shutdown_connections 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 >> 172.21.15.183:0/1335609681 conn(0x7f34800b4970 msgr2=0x7f34800af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 shutdown_connections 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 1 -- 172.21.15.183:0/1335609681 wait complete. 2024-03-29T10:51:54.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.272+0000 7f34863c8640 1 librados: shutdown 2024-03-29T10:51:54.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 9e754e8b-aaf9-4153-bc61-27aed626f0d4 3 2024-03-29T10:51:54.277 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.296+0000 7f4900421040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:51:54.293 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.311+0000 7f5afda2d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:51:54.331 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.350+0000 7f4900421040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:51:54.339 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.358+0000 7f5afda2d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:51:54.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.383+0000 7feb5ce9a640 10 monclient: get_monmap_and_config 2024-03-29T10:51:54.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.383+0000 7feb5ce9a640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.383+0000 7feb5ce9a640 10 monclient: monmap: 2024-03-29T10:51:54.364 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.364 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.364 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:54.383971+0000 2024-03-29T10:51:54.365 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:54.383971+0000 2024-03-29T10:51:54.365 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.365 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.365 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.365 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:54.365 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:54.365 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.383+0000 7feb5ce9a640 1 Processor -- start 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.383+0000 7feb5ce9a640 1 -- start start 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.383+0000 7feb5ce9a640 10 monclient: init 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 10 monclient(hunting): picked mon.noname-a con 0x7feb580b8240 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 10 monclient(hunting): picked mon.noname-c con 0x7feb580b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 10 monclient(hunting): picked mon.noname-b con 0x7feb5809c8c0 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5809bd80 con 0x7feb580b8240 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5809b960 con 0x7feb580b1f60 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580b2ec0 con 0x7feb5809c8c0 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb5ce9a640 10 monclient(hunting): authenticate will time out at 831.632400s 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb4ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7feb580b1f60 0x7feb5809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34752/0 (socket says 172.21.15.183:34752) 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb4ffff640 1 -- 172.21.15.183:0/745133718 learned_addr learned my addr 172.21.15.183:0/745133718 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50162863 0 0) 0x7feb5809b960 con 0x7feb580b1f60 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb44003520 con 0x7feb580b1f60 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726063490 0 0) 0x7feb580b2ec0 con 0x7feb5809c8c0 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb5809b960 con 0x7feb5809c8c0 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261509428 0 0) 0x7feb5809bd80 con 0x7feb580b8240 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.384+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb580b2ec0 con 0x7feb580b8240 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3294901281 0 0) 0x7feb44003520 con 0x7feb580b1f60 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 >> v1:172.21.15.155:6790/0 conn(0x7feb5809c8c0 legacy=0x7feb5814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 >> v1:172.21.15.155:6789/0 conn(0x7feb580b8240 legacy=0x7feb5813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 monclient: found mon.noname-c 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5813fb60 con 0x7feb580b1f60 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7feb40001690 con 0x7feb580b1f60 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: dump: 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.377 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb40001870 con 0x7feb580b1f60 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 1 -- 172.21.15.183:0/745133718 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7feb40001c80 con 0x7feb580b1f60 2024-03-29T10:51:54.378 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 4 set_mon_vals no callback set 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb55d9b640 10 monclient: dump: 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 10 monclient: shutdown 2024-03-29T10:51:54.379 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 1 -- 172.21.15.183:0/745133718 >> v1:172.21.15.183:6789/0 conn(0x7feb580b1f60 legacy=0x7feb5809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 1 -- 172.21.15.183:0/745133718 shutdown_connections 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 1 -- 172.21.15.183:0/745133718 >> 172.21.15.183:0/745133718 conn(0x7feb58066b00 msgr2=0x7feb58066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 1 -- 172.21.15.183:0/745133718 shutdown_connections 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.385+0000 7feb5ce9a640 1 -- 172.21.15.183:0/745133718 wait complete. 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.386+0000 7feb5ce9a640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.386+0000 7feb5ce9a640 10 monclient: monmap: 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.380 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.386+0000 7feb5ce9a640 1 librados: starting msgr at 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.386+0000 7feb5ce9a640 1 librados: starting objecter 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.386+0000 7feb5ce9a640 1 Processor -- start 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 1 -- start start 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 1 librados: setting wanted keys 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 1 librados: calling monclient init 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient: init 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient(hunting): picked mon.noname-a con 0x7feb5809c8c0 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.381 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient(hunting): picked mon.noname-b con 0x7feb580b8240 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient(hunting): picked mon.noname-c con 0x7feb581d4c20 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580afed0 con 0x7feb5809c8c0 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580aef20 con 0x7feb580b8240 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb581d3ff0 con 0x7feb581d4c20 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb5ce9a640 10 monclient(hunting): authenticate will time out at 831.635471s 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7feb580b8240 0x7feb581d4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34758/0 (socket says 172.21.15.183:34758) 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4ffff640 1 -- 172.21.15.183:0/2721560002 learned_addr learned my addr 172.21.15.183:0/2721560002 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 client.?.objecter ms_handle_connect 0x7feb580b8240 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 client.?.objecter ms_handle_connect 0x7feb5809c8c0 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.382 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 client.?.objecter ms_handle_connect 0x7feb581d4c20 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476979920 0 0) 0x7feb580aef20 con 0x7feb580b8240 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb38004eb0 con 0x7feb580b8240 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204164206 0 0) 0x7feb580afed0 con 0x7feb5809c8c0 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb580aef20 con 0x7feb5809c8c0 2024-03-29T10:51:54.383 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890878254 0 0) 0x7feb581d3ff0 con 0x7feb581d4c20 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.387+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb580afed0 con 0x7feb581d4c20 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3827453227 0 0) 0x7feb38004eb0 con 0x7feb580b8240 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb581d3ff0 con 0x7feb580b8240 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7feb40001b80 con 0x7feb580b8240 2024-03-29T10:51:54.384 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): dump: 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 76126741 0 0) 0x7feb580aef20 con 0x7feb5809c8c0 2024-03-29T10:51:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb38004eb0 con 0x7feb5809c8c0 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7feb480017d0 con 0x7feb5809c8c0 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): dump: 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.386 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2088589501 0 0) 0x7feb580afed0 con 0x7feb581d4c20 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb580aef20 con 0x7feb581d4c20 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7feb50001b60 con 0x7feb581d4c20 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient(hunting): dump: 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.387 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3461845515 0 0) 0x7feb581d3ff0 con 0x7feb580b8240 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 >> v1:172.21.15.155:6790/0 conn(0x7feb581d4c20 legacy=0x7feb581e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 >> v1:172.21.15.155:6789/0 conn(0x7feb5809c8c0 legacy=0x7feb580ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 monclient: found mon.b 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580b0640 con 0x7feb580b8240 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.388 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 10 monclient: _renew_subs 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb580adb00 con 0x7feb580b8240 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 10 monclient: _renew_subs 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb581e3360 con 0x7feb580b8240 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5ce9a640 1 librados: init done 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb40004bf0 con 0x7feb580b8240 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb5559a640 4 set_mon_vals no callback set 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7feb40004f00 con 0x7feb580b8240 2024-03-29T10:51:54.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 monclient: dump: 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feb400053f0 con 0x7feb580b8240 2024-03-29T10:51:54.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2187720915 0 0) 0x7feb40006a10 con 0x7feb580b8240 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 10 client.4123.objecter ms_dispatch 0x7feb58000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 3 client.4123.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 3 client.4123.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.388+0000 7feb4effd640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.389+0000 7feb5ce9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.389+0000 7feb5ce9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.389+0000 7feb5ce9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.389+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3c005080 con 0x7feb580b8240 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.391+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feb40006580 con 0x7feb580b8240 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.391+0000 7feb4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.391+0000 7feb4effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:54.392 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.395+0000 7f4900421040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:51:54.396 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.415+0000 7f5afda2d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:51:54.430 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.449+0000 7f4900421040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:51:54.440 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.458+0000 7f5afda2d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:51:54.443 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.462+0000 7feb5ce9a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9e754e8b-aaf9-4153-bc61-27aed626f0d4", "id": 3}] 2024-03-29T10:51:54.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.462+0000 7feb5ce9a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9e754e8b-aaf9-4153-bc61-27aed626f0d4", "id": 3}] 2024-03-29T10:51:54.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.462+0000 7feb5ce9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.462+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9e754e8b-aaf9-4153-bc61-27aed626f0d4", "id": 3} v 0) v1 -- 0x7feb3c0058b0 con 0x7feb580b8240 2024-03-29T10:51:54.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.466+0000 7feb4effd640 1 -- 172.21.15.183:0/2721560002 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e754e8b-aaf9-4153-bc61-27aed626f0d4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1727420490 0 525046306) 0x7feb40005830 con 0x7feb580b8240 2024-03-29T10:51:54.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.466+0000 7feb4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9e754e8b-aaf9-4153-bc61-27aed626f0d4", "id": 3}] 2024-03-29T10:51:54.448 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.466+0000 7feb4effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:54.448 INFO:teuthology.orchestra.run.smithi183.stdout:3 2024-03-29T10:51:54.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 10 librados: watch_flush enter 2024-03-29T10:51:54.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 10 librados: watch_flush exit 2024-03-29T10:51:54.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:51:54.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:51:54.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 10 monclient: shutdown 2024-03-29T10:51:54.449 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 >> v1:172.21.15.183:6789/0 conn(0x7feb580b8240 legacy=0x7feb581d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 shutdown_connections 2024-03-29T10:51:54.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 >> 172.21.15.183:0/2721560002 conn(0x7feb58066b00 msgr2=0x7feb580b7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 shutdown_connections 2024-03-29T10:51:54.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 1 -- 172.21.15.183:0/2721560002 wait complete. 2024-03-29T10:51:54.450 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.467+0000 7feb5ce9a640 1 librados: shutdown 2024-03-29T10:51:54.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 2d805188-e83a-4e5c-99b1-cedd6690d4f5 4 2024-03-29T10:51:54.473 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.492+0000 7f4900421040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:51:54.561 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.579+0000 7f5afda2d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:51:54.590 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.609+0000 7f4900421040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:51:54.600 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.617+0000 7fcc583d6640 10 monclient: get_monmap_and_config 2024-03-29T10:51:54.600 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.617+0000 7fcc583d6640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.600 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.617+0000 7fcc583d6640 10 monclient: monmap: 2024-03-29T10:51:54.600 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.600 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:54.619523+0000 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:54.619523+0000 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.618+0000 7fcc583d6640 1 Processor -- start 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.618+0000 7fcc583d6640 1 -- start start 2024-03-29T10:51:54.601 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.618+0000 7fcc583d6640 10 monclient: init 2024-03-29T10:51:54.612 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.618+0000 7fcc583d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.618+0000 7fcc583d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc5009ac40 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc5009c870 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc500b4360 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5009dec0 con 0x7fcc5009c870 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5009da30 con 0x7fcc5009ac40 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500b39c0 con 0x7fcc500b4360 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 10 monclient(hunting): authenticate will time out at 831.868030s 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc56172640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fcc5009ac40 0x7fcc500b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34760/0 (socket says 172.21.15.183:34760) 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc56172640 1 -- 172.21.15.183:0/998521572 learned_addr learned my addr 172.21.15.183:0/998521572 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679297207 0 0) 0x7fcc5009da30 con 0x7fcc5009ac40 2024-03-29T10:51:54.613 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c003520 con 0x7fcc5009ac40 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991263883 0 0) 0x7fcc500b39c0 con 0x7fcc500b4360 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc5009da30 con 0x7fcc500b4360 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539246299 0 0) 0x7fcc40001110 con 0x7fcc5009c870 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.614 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.615 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:51:54.615 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc500b39c0 con 0x7fcc5009c870 2024-03-29T10:51:54.615 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 479706080 0 0) 0x7fcc3c003520 con 0x7fcc5009ac40 2024-03-29T10:51:54.615 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 >> v1:172.21.15.155:6790/0 conn(0x7fcc500b4360 legacy=0x7fcc5014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.615 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 >> v1:172.21.15.155:6789/0 conn(0x7fcc5009c870 legacy=0x7fcc5013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.615 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 monclient: found mon.noname-c 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500b5160 con 0x7fcc5009ac40 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcc4c001690 con 0x7fcc5009ac40 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: dump: 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc4c001870 con 0x7fcc5009ac40 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.617 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 1 -- 172.21.15.183:0/998521572 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcc4c001c80 con 0x7fcc5009ac40 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc583d6640 4 set_mon_vals no callback set 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.619+0000 7fcc55170640 10 monclient: dump: 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.618 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.620+0000 7fcc583d6640 10 monclient: shutdown 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.620+0000 7fcc583d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.620+0000 7fcc583d6640 1 -- 172.21.15.183:0/998521572 >> v1:172.21.15.183:6789/0 conn(0x7fcc5009ac40 legacy=0x7fcc500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.620+0000 7fcc583d6640 1 -- 172.21.15.183:0/998521572 shutdown_connections 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.620+0000 7fcc583d6640 1 -- 172.21.15.183:0/998521572 >> 172.21.15.183:0/998521572 conn(0x7fcc50066b00 msgr2=0x7fcc50066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.620+0000 7fcc583d6640 1 -- 172.21.15.183:0/998521572 shutdown_connections 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.620+0000 7fcc583d6640 1 -- 172.21.15.183:0/998521572 wait complete. 2024-03-29T10:51:54.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient: monmap: 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:54.620 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 librados: starting msgr at 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 librados: starting objecter 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 Processor -- start 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 -- start start 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 librados: setting wanted keys 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 librados: calling monclient init 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient: init 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcc5009ac40 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient(hunting): picked mon.noname-b con 0x7fcc50138f60 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcc500afdc0 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc501db610 con 0x7fcc5009ac40 2024-03-29T10:51:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc50139a00 con 0x7fcc50138f60 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc500b0230 con 0x7fcc500afdc0 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.621+0000 7fcc583d6640 10 monclient(hunting): authenticate will time out at 831.870938s 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc55971640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fcc50138f60 0x7fcc500af6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34764/0 (socket says 172.21.15.183:34764) 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc55971640 1 -- 172.21.15.183:0/3578092618 learned_addr learned my addr 172.21.15.183:0/3578092618 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 client.?.objecter ms_handle_connect 0x7fcc50138f60 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 client.?.objecter ms_handle_connect 0x7fcc5009ac40 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.622 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 client.?.objecter ms_handle_connect 0x7fcc500afdc0 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907087433 0 0) 0x7fcc50139a00 con 0x7fcc50138f60 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc34004eb0 con 0x7fcc50138f60 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841147892 0 0) 0x7fcc500b0230 con 0x7fcc500afdc0 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.623 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc50139a00 con 0x7fcc500afdc0 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879294741 0 0) 0x7fcc501db610 con 0x7fcc5009ac40 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc500b0230 con 0x7fcc5009ac40 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2993918944 0 0) 0x7fcc34004eb0 con 0x7fcc50138f60 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc501db610 con 0x7fcc50138f60 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcc400017e0 con 0x7fcc50138f60 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:54.624 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.625 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264283660 0 0) 0x7fcc50139a00 con 0x7fcc500afdc0 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc34004eb0 con 0x7fcc500afdc0 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcc48001b50 con 0x7fcc500afdc0 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.626 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3127806523 0 0) 0x7fcc500b0230 con 0x7fcc5009ac40 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc50139a00 con 0x7fcc5009ac40 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcc4c001b80 con 0x7fcc5009ac40 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.627 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3292133088 0 0) 0x7fcc501db610 con 0x7fcc50138f60 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 >> v1:172.21.15.155:6790/0 conn(0x7fcc500afdc0 legacy=0x7fcc501e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 >> v1:172.21.15.155:6789/0 conn(0x7fcc5009ac40 legacy=0x7fcc501dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 monclient: found mon.b 2024-03-29T10:51:54.628 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.629 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.629 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc500b03c0 con 0x7fcc50138f60 2024-03-29T10:51:54.629 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.629 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.629 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.629 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:51:54.629 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 10 monclient: _renew_subs 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc501e36e0 con 0x7fcc50138f60 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 10 monclient: _renew_subs 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc501e3a50 con 0x7fcc50138f60 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc583d6640 1 librados: init done 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc400019c0 con 0x7fcc50138f60 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcc40003610 con 0x7fcc50138f60 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.630 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 monclient: dump: 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc5496f640 4 set_mon_vals no callback set 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcc40003bc0 con 0x7fcc50138f60 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3585135954 0 0) 0x7fcc40006490 con 0x7fcc50138f60 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 10 client.4129.objecter ms_dispatch 0x7fcc50000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:51:54.631 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 3 client.4129.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.622+0000 7fcc46ffd640 3 client.4129.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.623+0000 7fcc46ffd640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.624+0000 7fcc583d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.624+0000 7fcc583d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.624+0000 7fcc583d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.624+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc38005080 con 0x7fcc50138f60 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.626+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcc400057c0 con 0x7fcc50138f60 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.626+0000 7fcc46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.632 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.626+0000 7fcc46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:54.654 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.672+0000 7f5afda2d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:54.660 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.679+0000 7f4900421040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:54.700 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.718+0000 7f5afda2d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:51:54.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.719+0000 7fcc583d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2d805188-e83a-4e5c-99b1-cedd6690d4f5", "id": 4}] 2024-03-29T10:51:54.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.719+0000 7fcc583d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2d805188-e83a-4e5c-99b1-cedd6690d4f5", "id": 4}] 2024-03-29T10:51:54.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.719+0000 7fcc583d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.719+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2d805188-e83a-4e5c-99b1-cedd6690d4f5", "id": 4} v 0) v1 -- 0x7fcc380058b0 con 0x7fcc50138f60 2024-03-29T10:51:54.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.723+0000 7fcc46ffd640 1 -- 172.21.15.183:0/3578092618 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2d805188-e83a-4e5c-99b1-cedd6690d4f5", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2314315711 0 1697012583) 0x7fcc400040f0 con 0x7fcc50138f60 2024-03-29T10:51:54.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.723+0000 7fcc46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2d805188-e83a-4e5c-99b1-cedd6690d4f5", "id": 4}] 2024-03-29T10:51:54.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.723+0000 7fcc46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:54.706 INFO:teuthology.orchestra.run.smithi183.stdout:4 2024-03-29T10:51:54.707 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 10 librados: watch_flush enter 2024-03-29T10:51:54.707 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 10 librados: watch_flush exit 2024-03-29T10:51:54.707 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 10 monclient: shutdown 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 >> v1:172.21.15.183:6789/0 conn(0x7fcc50138f60 legacy=0x7fcc500af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 shutdown_connections 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 >> 172.21.15.183:0/3578092618 conn(0x7fcc50066b00 msgr2=0x7fcc500b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 shutdown_connections 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 1 -- 172.21.15.183:0/3578092618 wait complete. 2024-03-29T10:51:54.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.724+0000 7fcc583d6640 1 librados: shutdown 2024-03-29T10:51:54.708 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.726+0000 7f4900421040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:51:54.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8 5 2024-03-29T10:51:54.757 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.774+0000 7f5afda2d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:51:54.760 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.779+0000 7f4900421040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:51:54.805 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:54.823+0000 7f5afda2d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:51:54.806 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:54.826+0000 7f4900421040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:51:54.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.833+0000 7f07d16f8640 10 monclient: get_monmap_and_config 2024-03-29T10:51:54.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.833+0000 7f07d16f8640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.833+0000 7f07d16f8640 10 monclient: monmap: 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:54.835198+0000 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:54.835198+0000 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:54.816 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.833+0000 7f07d16f8640 1 Processor -- start 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.833+0000 7f07d16f8640 1 -- start start 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.833+0000 7f07d16f8640 10 monclient: init 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 10 monclient(hunting): picked mon.noname-c con 0x7f07cc09ac40 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 10 monclient(hunting): picked mon.noname-b con 0x7f07cc09c870 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 10 monclient(hunting): picked mon.noname-a con 0x7f07cc0b4360 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc09dec0 con 0x7f07cc0b4360 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc09da30 con 0x7f07cc09ac40 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc0b39c0 con 0x7f07cc09c870 2024-03-29T10:51:54.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07d16f8640 10 monclient(hunting): authenticate will time out at 832.083528s 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07caffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f07cc09ac40 0x7f07cc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34768/0 (socket says 172.21.15.183:34768) 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07caffd640 1 -- 172.21.15.183:0/1627587610 learned_addr learned my addr 172.21.15.183:0/1627587610 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845437387 0 0) 0x7f07cc09da30 con 0x7f07cc09ac40 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07ca7fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.834+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07b8003520 con 0x7f07cc09ac40 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848981610 0 0) 0x7f07c4001490 con 0x7f07cc0b4360 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.826 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc09da30 con 0x7f07cc0b4360 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275189954 0 0) 0x7f07cc0b39c0 con 0x7f07cc09c870 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c4001490 con 0x7f07cc09c870 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43172577 0 0) 0x7f07b8003520 con 0x7f07cc09ac40 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 >> v1:172.21.15.155:6790/0 conn(0x7f07cc09c870 legacy=0x7f07cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 >> v1:172.21.15.155:6789/0 conn(0x7f07cc0b4360 legacy=0x7f07cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 monclient: found mon.noname-c 2024-03-29T10:51:54.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc0b5160 con 0x7f07cc09ac40 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f07bc001690 con 0x7f07cc09ac40 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: dump: 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.828 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07bc001870 con 0x7f07cc09ac40 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 1 -- 172.21.15.183:0/1627587610 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f07bc001c80 con 0x7f07cc09ac40 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 4 set_mon_vals no callback set 2024-03-29T10:51:54.829 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07ca7fc640 10 monclient: dump: 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 10 monclient: shutdown 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.830 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 1 -- 172.21.15.183:0/1627587610 >> v1:172.21.15.183:6789/0 conn(0x7f07cc09ac40 legacy=0x7f07cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 1 -- 172.21.15.183:0/1627587610 shutdown_connections 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 1 -- 172.21.15.183:0/1627587610 >> 172.21.15.183:0/1627587610 conn(0x7f07cc066b00 msgr2=0x7f07cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 1 -- 172.21.15.183:0/1627587610 shutdown_connections 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.835+0000 7f07d16f8640 1 -- 172.21.15.183:0/1627587610 wait complete. 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.836+0000 7f07d16f8640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.836+0000 7f07d16f8640 10 monclient: monmap: 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:54.831 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.836+0000 7f07d16f8640 1 librados: starting msgr at 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.836+0000 7f07d16f8640 1 librados: starting objecter 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.836+0000 7f07d16f8640 1 Processor -- start 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 1 -- start start 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 1 librados: setting wanted keys 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 1 librados: calling monclient init 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient: init 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient(hunting): picked mon.noname-c con 0x7f07cc09ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient(hunting): picked mon.noname-a con 0x7f07cc0b1860 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient(hunting): picked mon.noname-b con 0x7f07cc13b5b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:54.832 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc1d7660 con 0x7f07cc0b1860 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc0ade20 con 0x7f07cc13b5b0 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc1406b0 con 0x7f07cc09ac40 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07d16f8640 10 monclient(hunting): authenticate will time out at 832.086641s 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07cb7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f07cc13b5b0 0x7f07cc1e2830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34778/0 (socket says 172.21.15.183:34778) 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07cb7fe640 1 -- 172.21.15.183:0/1929816570 learned_addr learned my addr 172.21.15.183:0/1929816570 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07c3fff640 10 client.?.objecter ms_handle_connect 0x7f07cc13b5b0 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07c3fff640 10 client.?.objecter ms_handle_connect 0x7f07cc0b1860 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07c3fff640 10 client.?.objecter ms_handle_connect 0x7f07cc09ac40 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.837+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681367681 0 0) 0x7f07cc0ade20 con 0x7f07cc13b5b0 2024-03-29T10:51:54.833 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac004eb0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163591625 0 0) 0x7f07cc1d7660 con 0x7f07cc0b1860 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc0ade20 con 0x7f07cc0b1860 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629784746 0 0) 0x7f07cc1406b0 con 0x7f07cc09ac40 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.834 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07cc1d7660 con 0x7f07cc09ac40 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707532472 0 0) 0x7f07ac004eb0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07cc1406b0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f07c4001c80 con 0x7f07cc13b5b0 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): dump: 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482955528 0 0) 0x7f07cc0ade20 con 0x7f07cc0b1860 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07ac004eb0 con 0x7f07cc0b1860 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1406487598 0 0) 0x7f07cc1d7660 con 0x7f07cc09ac40 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07cc0ade20 con 0x7f07cc09ac40 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f07b4001720 con 0x7f07cc0b1860 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:54.836 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): dump: 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f07bc0019e0 con 0x7f07cc09ac40 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:54.837 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient(hunting): dump: 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1464710357 0 0) 0x7f07cc1406b0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 >> v1:172.21.15.155:6790/0 conn(0x7f07cc09ac40 legacy=0x7f07cc13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 >> v1:172.21.15.155:6789/0 conn(0x7f07cc0b1860 legacy=0x7f07cc13aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 monclient: found mon.b 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc1cccd0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 10 monclient: _renew_subs 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc1cce60 con 0x7f07cc13b5b0 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 10 monclient: _renew_subs 2024-03-29T10:51:54.839 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc1d7190 con 0x7f07cc13b5b0 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07d16f8640 1 librados: init done 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07c4003970 con 0x7f07cc13b5b0 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f07c4003d80 con 0x7f07cc13b5b0 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 monclient: dump: 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f07c40042f0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c9ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3166222275 0 0) 0x7f07c40045a0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 10 client.4135.objecter ms_dispatch 0x7f07cc000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 3 client.4135.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 3 client.4135.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.838+0000 7f07c3fff640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.839+0000 7f07d16f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.839+0000 7f07d16f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.839+0000 7f07d16f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.841 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.839+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b0005080 con 0x7f07cc13b5b0 2024-03-29T10:51:54.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.841+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f07c4005b40 con 0x7f07cc13b5b0 2024-03-29T10:51:54.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.841+0000 7f07c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.842 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.841+0000 7f07c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:54.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.912+0000 7f07d16f8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8", "id": 5}] 2024-03-29T10:51:54.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.912+0000 7f07d16f8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8", "id": 5}] 2024-03-29T10:51:54.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.912+0000 7f07d16f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:54.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.912+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8", "id": 5} v 0) v1 -- 0x7f07b00058b0 con 0x7f07cc13b5b0 2024-03-29T10:51:54.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.915+0000 7f07c3fff640 1 -- 172.21.15.183:0/1929816570 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1294059206 0 1988412176) 0x7f07c4005d60 con 0x7f07cc13b5b0 2024-03-29T10:51:54.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8", "id": 5}] 2024-03-29T10:51:54.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:54.898 INFO:teuthology.orchestra.run.smithi183.stdout:5 2024-03-29T10:51:54.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 10 librados: watch_flush enter 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 10 librados: watch_flush exit 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 10 client.4135.objecter successfully canceled tick 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 10 monclient: shutdown 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 >> v1:172.21.15.183:6789/0 conn(0x7f07cc13b5b0 legacy=0x7f07cc1e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 shutdown_connections 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.916+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 >> 172.21.15.183:0/1929816570 conn(0x7f07cc066b00 msgr2=0x7f07cc13cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.917+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 shutdown_connections 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.917+0000 7f07d16f8640 1 -- 172.21.15.183:0/1929816570 wait complete. 2024-03-29T10:51:54.900 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:54.917+0000 7f07d16f8640 1 librados: shutdown 2024-03-29T10:51:54.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8 6 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.067+0000 7f351b89a640 10 monclient: get_monmap_and_config 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.067+0000 7f351b89a640 10 monclient: build_initial_monmap 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.067+0000 7f351b89a640 10 monclient: monmap: 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:55.069121+0000 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:55.069121+0000 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:55.050 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:55.051 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:55.051 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.067+0000 7f351b89a640 1 Processor -- start 2024-03-29T10:51:55.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 1 -- start start 2024-03-29T10:51:55.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient: init 2024-03-29T10:51:55.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:55.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:55.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient(hunting): picked mon.noname-b con 0x7f351409ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:55.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient(hunting): picked mon.noname-a con 0x7f351409c870 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:55.060 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient(hunting): picked mon.noname-c con 0x7f35140b4360 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351409dec0 con 0x7f351409c870 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351409da30 con 0x7f35140b4360 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35140b39c0 con 0x7f351409ac40 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f351b89a640 10 monclient(hunting): authenticate will time out at 832.317577s 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f3519e37640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f35140b4360 0x7f351414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34792/0 (socket says 172.21.15.183:34792) 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f3519e37640 1 -- 172.21.15.183:0/2546374065 learned_addr learned my addr 172.21.15.183:0/2546374065 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837248297 0 0) 0x7f351409da30 con 0x7f35140b4360 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f350bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f350bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.068+0000 7f350bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3500003520 con 0x7f35140b4360 2024-03-29T10:51:55.061 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666844997 0 0) 0x7f3510001110 con 0x7f351409c870 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f351409da30 con 0x7f351409c870 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345819458 0 0) 0x7f35140b39c0 con 0x7f351409ac40 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3510001110 con 0x7f351409ac40 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682436417 0 0) 0x7f3500003520 con 0x7f35140b4360 2024-03-29T10:51:55.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 >> v1:172.21.15.155:6790/0 conn(0x7f351409ac40 legacy=0x7f35140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 >> v1:172.21.15.155:6789/0 conn(0x7f351409c870 legacy=0x7f351413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 monclient: found mon.noname-c 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35140b5160 con 0x7f35140b4360 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: _finish_auth 0 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f350c001a10 con 0x7f35140b4360 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f351b89a640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f351b89a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:55.063 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: dump: 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f351b89a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f350c001bf0 con 0x7f35140b4360 2024-03-29T10:51:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 1 -- 172.21.15.183:0/2546374065 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f350c002000 con 0x7f35140b4360 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f350bfff640 10 monclient: dump: 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.065 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f351b89a640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.069+0000 7f351b89a640 4 set_mon_vals no callback set 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.070+0000 7f351b89a640 10 monclient: shutdown 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.070+0000 7f351b89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.070+0000 7f351b89a640 1 -- 172.21.15.183:0/2546374065 >> v1:172.21.15.183:6789/0 conn(0x7f35140b4360 legacy=0x7f351414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.070+0000 7f351b89a640 1 -- 172.21.15.183:0/2546374065 shutdown_connections 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.070+0000 7f351b89a640 1 -- 172.21.15.183:0/2546374065 >> 172.21.15.183:0/2546374065 conn(0x7f3514066b00 msgr2=0x7f3514066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.070+0000 7f351b89a640 1 -- 172.21.15.183:0/2546374065 shutdown_connections 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.070+0000 7f351b89a640 1 -- 172.21.15.183:0/2546374065 wait complete. 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 10 monclient: build_initial_monmap 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 10 monclient: monmap: 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:55.066 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 1 librados: starting msgr at 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 1 librados: starting objecter 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 1 Processor -- start 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 1 -- start start 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 1 librados: setting wanted keys 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 1 librados: calling monclient init 2024-03-29T10:51:55.067 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.071+0000 7f351b89a640 10 monclient: init 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 10 monclient(hunting): picked mon.noname-c con 0x7f351409ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 10 monclient(hunting): picked mon.noname-a con 0x7f35141d7cd0 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 10 monclient(hunting): picked mon.noname-b con 0x7f35140ad1e0 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35141d7130 con 0x7f35141d7cd0 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3514139b10 con 0x7f35140ad1e0 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351413ffa0 con 0x7f351409ac40 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f351b89a640 10 monclient(hunting): authenticate will time out at 832.321715s 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f3519e37640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f35140ad1e0 0x7f3514139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34796/0 (socket says 172.21.15.183:34796) 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f3519e37640 1 -- 172.21.15.183:0/1029335411 learned_addr learned my addr 172.21.15.183:0/1029335411 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:55.068 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f3509ffb640 10 client.?.objecter ms_handle_connect 0x7f35140ad1e0 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.072+0000 7f3509ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 client.?.objecter ms_handle_connect 0x7f351409ac40 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 client.?.objecter ms_handle_connect 0x7f35141d7cd0 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602644281 0 0) 0x7f3514139b10 con 0x7f35140ad1e0 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f8004eb0 con 0x7f35140ad1e0 2024-03-29T10:51:55.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891317326 0 0) 0x7f35141d7130 con 0x7f35141d7cd0 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3514139b10 con 0x7f35141d7cd0 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116021405 0 0) 0x7f351413ffa0 con 0x7f351409ac40 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.070 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35141d7130 con 0x7f351409ac40 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1982498114 0 0) 0x7f34f8004eb0 con 0x7f35140ad1e0 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f351413ffa0 con 0x7f35140ad1e0 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f350c001f40 con 0x7f35140ad1e0 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.071 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1661178924 0 0) 0x7f3514139b10 con 0x7f35141d7cd0 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34f8004eb0 con 0x7f35141d7cd0 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3510001920 con 0x7f35141d7cd0 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.072 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2880570450 0 0) 0x7f35141d7130 con 0x7f351409ac40 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3514139b10 con 0x7f351409ac40 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3504001870 con 0x7f351409ac40 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3083782037 0 0) 0x7f351413ffa0 con 0x7f35140ad1e0 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 >> v1:172.21.15.155:6790/0 conn(0x7f351409ac40 legacy=0x7f35141d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 >> v1:172.21.15.155:6789/0 conn(0x7f35141d7cd0 legacy=0x7f3514138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 monclient: found mon.b 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35140ae520 con 0x7f35140ad1e0 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 10 monclient: _renew_subs 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35140ae800 con 0x7f35140ad1e0 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f350c004e10 con 0x7f35140ad1e0 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:55.075 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f350c005120 con 0x7f35140ad1e0 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 10 monclient: dump: 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f350c005690 con 0x7f35140ad1e0 2024-03-29T10:51:55.076 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f350b7fe640 4 set_mon_vals no callback set 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 10 monclient: _renew_subs 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3514142b70 con 0x7f35140ad1e0 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.073+0000 7f351b89a640 1 librados: init done 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1932739522 0 0) 0x7f350c0061b0 con 0x7f35140ad1e0 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f3509ffb640 10 client.4141.objecter ms_dispatch 0x7f3514000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f3509ffb640 3 client.4141.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f3509ffb640 3 client.4141.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f3509ffb640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f351b89a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f351b89a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f351b89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.077 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.074+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351409b020 con 0x7f35140ad1e0 2024-03-29T10:51:55.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.076+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f350c006370 con 0x7f35140ad1e0 2024-03-29T10:51:55.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.076+0000 7f3509ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.076+0000 7f3509ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:55.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.147+0000 7f351b89a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8", "id": 6}] 2024-03-29T10:51:55.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.147+0000 7f351b89a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8", "id": 6}] 2024-03-29T10:51:55.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.147+0000 7f351b89a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.130 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.147+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8", "id": 6} v 0) v1 -- 0x7f351413df40 con 0x7f35140ad1e0 2024-03-29T10:51:55.134 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.151+0000 7f3509ffb640 1 -- 172.21.15.183:0/1029335411 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3728352895 0 1113811849) 0x7f350c005c90 con 0x7f35140ad1e0 2024-03-29T10:51:55.134 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.151+0000 7f3509ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8", "id": 6}] 2024-03-29T10:51:55.134 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.151+0000 7f3509ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:55.134 INFO:teuthology.orchestra.run.smithi183.stdout:6 2024-03-29T10:51:55.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 10 librados: watch_flush enter 2024-03-29T10:51:55.135 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 10 librados: watch_flush exit 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 10 monclient: shutdown 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 >> v1:172.21.15.183:6789/0 conn(0x7f35140ad1e0 legacy=0x7f3514139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 shutdown_connections 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 >> 172.21.15.183:0/1029335411 conn(0x7f3514066b00 msgr2=0x7f35140b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 shutdown_connections 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.152+0000 7f351b89a640 1 -- 172.21.15.183:0/1029335411 wait complete. 2024-03-29T10:51:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.153+0000 7f351b89a640 1 librados: shutdown 2024-03-29T10:51:55.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new c9a1875d-6ecd-4408-ae84-597c695ec1af 7 2024-03-29T10:51:55.194 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:55.213+0000 7f4900421040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:51:55.203 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.221+0000 7f5afda2d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:51:55.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.263+0000 7f9d28590640 10 monclient: get_monmap_and_config 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.263+0000 7f9d28590640 10 monclient: build_initial_monmap 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.263+0000 7f9d28590640 10 monclient: monmap: 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:55.264883+0000 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:55.264883+0000 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.263+0000 7f9d28590640 1 Processor -- start 2024-03-29T10:51:55.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.263+0000 7f9d28590640 1 -- start start 2024-03-29T10:51:55.247 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.263+0000 7f9d28590640 10 monclient: init 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 10 monclient(hunting): picked mon.noname-a con 0x7f9d2009ac40 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 10 monclient(hunting): picked mon.noname-c con 0x7f9d2009c870 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 10 monclient(hunting): picked mon.noname-b con 0x7f9d200b4360 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2009dec0 con 0x7f9d2009ac40 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2009da30 con 0x7f9d2009c870 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d200b39c0 con 0x7f9d200b4360 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d28590640 10 monclient(hunting): authenticate will time out at 832.513304s 2024-03-29T10:51:55.255 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d25b2b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f9d2009c870 0x7f9d2013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34798/0 (socket says 172.21.15.183:34798) 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d25b2b640 1 -- 172.21.15.183:0/464595326 learned_addr learned my addr 172.21.15.183:0/464595326 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166833115 0 0) 0x7f9d2009da30 con 0x7f9d2009c870 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d0c003520 con 0x7f9d2009c870 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190487691 0 0) 0x7f9d1c001040 con 0x7f9d2009ac40 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:51:55.256 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2009da30 con 0x7f9d2009ac40 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424326776 0 0) 0x7f9d200b39c0 con 0x7f9d200b4360 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d1c001040 con 0x7f9d200b4360 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615086865 0 0) 0x7f9d0c003520 con 0x7f9d2009c870 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 >> v1:172.21.15.155:6790/0 conn(0x7f9d200b4360 legacy=0x7f9d2014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.264+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 >> v1:172.21.15.155:6789/0 conn(0x7f9d2009ac40 legacy=0x7f9d200b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: _finish_hunting 0 2024-03-29T10:51:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 1 monclient: found mon.noname-c 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d200b5160 con 0x7f9d2009c870 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: _finish_auth 0 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: _check_auth_tickets 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9d10001690 con 0x7f9d2009c870 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: dump: 2024-03-29T10:51:55.258 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d10001870 con 0x7f9d2009c870 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 1 -- 172.21.15.183:0/464595326 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9d10001c80 con 0x7f9d2009c870 2024-03-29T10:51:55.259 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 4 set_mon_vals no callback set 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d2532a640 10 monclient: dump: 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 10 monclient: shutdown 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 1 -- 172.21.15.183:0/464595326 >> v1:172.21.15.183:6789/0 conn(0x7f9d2009c870 legacy=0x7f9d2013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 1 -- 172.21.15.183:0/464595326 shutdown_connections 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 1 -- 172.21.15.183:0/464595326 >> 172.21.15.183:0/464595326 conn(0x7f9d20066b00 msgr2=0x7f9d20066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 1 -- 172.21.15.183:0/464595326 shutdown_connections 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.265+0000 7f9d28590640 1 -- 172.21.15.183:0/464595326 wait complete. 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 10 monclient: build_initial_monmap 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 10 monclient: monmap: 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-29T10:51:55.261 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 1 librados: starting msgr at 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 1 librados: starting objecter 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 1 Processor -- start 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 1 -- start start 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 1 librados: setting wanted keys 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 1 librados: calling monclient init 2024-03-29T10:51:55.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.266+0000 7f9d28590640 10 monclient: init 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 10 monclient(hunting): picked mon.noname-c con 0x7f9d2009ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 10 monclient(hunting): picked mon.noname-a con 0x7f9d2013ffa0 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 10 monclient(hunting): picked mon.noname-b con 0x7f9d201409b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2013a7f0 con 0x7f9d2013ffa0 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d20140600 con 0x7f9d201409b0 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d201d78d0 con 0x7f9d2009ac40 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d28590640 10 monclient(hunting): authenticate will time out at 832.516368s 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d26b2d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f9d201409b0 0x7f9d2013a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:34812/0 (socket says 172.21.15.183:34812) 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d26b2d640 1 -- 172.21.15.183:0/1510323632 learned_addr learned my addr 172.21.15.183:0/1510323632 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 client.?.objecter ms_handle_connect 0x7f9d2009ac40 2024-03-29T10:51:55.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 client.?.objecter ms_handle_connect 0x7f9d201409b0 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 client.?.objecter ms_handle_connect 0x7f9d2013ffa0 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129978308 0 0) 0x7f9d20140600 con 0x7f9d201409b0 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d04004eb0 con 0x7f9d201409b0 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994840602 0 0) 0x7f9d2013a7f0 con 0x7f9d2013ffa0 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.264 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d20140600 con 0x7f9d2013ffa0 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011726787 0 0) 0x7f9d201d78d0 con 0x7f9d2009ac40 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 10 monclient(hunting): my global_id is 4160 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2013a7f0 con 0x7f9d2009ac40 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 189175527 0 0) 0x7f9d04004eb0 con 0x7f9d201409b0 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.267+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d201d78d0 con 0x7f9d201409b0 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9d18001be0 con 0x7f9d201409b0 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831134161 0 0) 0x7f9d20140600 con 0x7f9d2013ffa0 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d04004eb0 con 0x7f9d2013ffa0 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1562232610 0 0) 0x7f9d2013a7f0 con 0x7f9d2009ac40 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d20140600 con 0x7f9d2009ac40 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9d100019e0 con 0x7f9d2013ffa0 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.267 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9d1c0017d0 con 0x7f9d2009ac40 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.268 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1784726841 0 0) 0x7f9d201d78d0 con 0x7f9d201409b0 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 >> v1:172.21.15.155:6790/0 conn(0x7f9d2009ac40 legacy=0x7f9d201d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 >> v1:172.21.15.155:6789/0 conn(0x7f9d2013ffa0 legacy=0x7f9d201399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 monclient: found mon.b 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d201427d0 con 0x7f9d201409b0 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 10 monclient: _renew_subs 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d20142a80 con 0x7f9d201409b0 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 10 monclient: _renew_subs 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d200b80a0 con 0x7f9d201409b0 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d28590640 1 librados: init done 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d18003990 con 0x7f9d201409b0 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9d18003ce0 con 0x7f9d201409b0 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 monclient: dump: 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-29T10:51:55.270 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d24b29640 4 set_mon_vals no callback set 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9d18004290 con 0x7f9d201409b0 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1902907968 0 0) 0x7f9d18006b90 con 0x7f9d201409b0 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 10 client.4147.objecter ms_dispatch 0x7f9d20000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 3 client.4147.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:51:55.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 3 client.4147.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.268+0000 7f9d16ffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.269+0000 7f9d28590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.269+0000 7f9d28590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.269+0000 7f9d28590640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.269+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d08005080 con 0x7f9d201409b0 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.271+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9d18004770 con 0x7f9d201409b0 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.271+0000 7f9d16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.272 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.271+0000 7f9d16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:55.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.343+0000 7f9d28590640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c9a1875d-6ecd-4408-ae84-597c695ec1af", "id": 7}] 2024-03-29T10:51:55.325 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.343+0000 7f9d28590640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c9a1875d-6ecd-4408-ae84-597c695ec1af", "id": 7}] 2024-03-29T10:51:55.326 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.343+0000 7f9d28590640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:55.326 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.343+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c9a1875d-6ecd-4408-ae84-597c695ec1af", "id": 7} v 0) v1 -- 0x7f9d080058b0 con 0x7f9d201409b0 2024-03-29T10:51:55.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.346+0000 7f9d16ffd640 1 -- 172.21.15.183:0/1510323632 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c9a1875d-6ecd-4408-ae84-597c695ec1af", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1996650843 0 1371665406) 0x7f9d18006530 con 0x7f9d201409b0 2024-03-29T10:51:55.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.346+0000 7f9d16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c9a1875d-6ecd-4408-ae84-597c695ec1af", "id": 7}] 2024-03-29T10:51:55.328 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.346+0000 7f9d16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:55.329 INFO:teuthology.orchestra.run.smithi183.stdout:7 2024-03-29T10:51:55.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 10 librados: watch_flush enter 2024-03-29T10:51:55.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 10 librados: watch_flush exit 2024-03-29T10:51:55.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:51:55.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:51:55.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 10 monclient: shutdown 2024-03-29T10:51:55.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:55.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 >> v1:172.21.15.183:6789/0 conn(0x7f9d201409b0 legacy=0x7f9d2013a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 shutdown_connections 2024-03-29T10:51:55.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 >> 172.21.15.183:0/1510323632 conn(0x7f9d20066b00 msgr2=0x7f9d2013e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:55.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 shutdown_connections 2024-03-29T10:51:55.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 1 -- 172.21.15.183:0/1510323632 wait complete. 2024-03-29T10:51:55.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-29T10:51:55.347+0000 7f9d28590640 1 librados: shutdown 2024-03-29T10:51:55.335 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:55.350+0000 7f4900421040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:51:55.340 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:51:55.340 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:51:55.342 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:51:55.342 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:51:55.342 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:51:55.344 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:51:55.345 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:51:55.345 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:51:55.345 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.362+0000 7f5afda2d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:51:55.347 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:51:55.347 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:51:55.347 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:51:55.349 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:51:55.349 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:51:55.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:51:55.351 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:51:55.351 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:51:55.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:51:55.353 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:51:55.353 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:51:55.354 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:51:55.355 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:51:55.356 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:51:55.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:51:55.358 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:51:55.358 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:55.480 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.498+0000 7f5afda2d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.521+0000 7f96458a8640 10 monclient: get_monmap_and_config 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.521+0000 7f96458a8640 10 monclient: build_initial_monmap 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.521+0000 7f96458a8640 10 monclient: monmap: 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:55.522578+0000 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:55.522578+0000 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:55.503 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.522+0000 7f96458a8640 1 Processor -- start 2024-03-29T10:51:55.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.522+0000 7f96458a8640 1 -- start start 2024-03-29T10:51:55.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.522+0000 7f96458a8640 10 monclient: init 2024-03-29T10:51:55.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:55.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:55.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient(hunting): picked mon.noname-b con 0x7f964009ac40 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient(hunting): picked mon.noname-a con 0x7f964009c870 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient(hunting): picked mon.noname-c con 0x7f96400b4360 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964009dec0 con 0x7f964009c870 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964009da30 con 0x7f96400b4360 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96400b39c0 con 0x7f964009ac40 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient(hunting): authenticate will time out at 818.815028s 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963e7fc640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f964009c870 0x7f964013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50020/0 (socket says 172.21.15.155:50020) 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963e7fc640 1 -- 172.21.15.155:0/1081059649 learned_addr learned my addr 172.21.15.155:0/1081059649 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613583960 0 0) 0x7f9630001110 con 0x7f964009c870 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9628003520 con 0x7f964009c870 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130828024 0 0) 0x7f96400b39c0 con 0x7f964009ac40 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9630001110 con 0x7f964009ac40 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290180303 0 0) 0x7f964009da30 con 0x7f96400b4360 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient(hunting): my global_id is 4174 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96400b39c0 con 0x7f96400b4360 2024-03-29T10:51:55.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23479929 0 0) 0x7f9630001110 con 0x7f964009ac40 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 >> v1:172.21.15.183:6789/0 conn(0x7f96400b4360 legacy=0x7f964014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 >> v1:172.21.15.155:6789/0 conn(0x7f964009c870 legacy=0x7f964013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 monclient: found mon.noname-b 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96400b5160 con 0x7f964009ac40 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9634001670 con 0x7f964009ac40 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 5 monclient: authenticate success, global_id 4187 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:55.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: dump: 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.516 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9634001850 con 0x7f964009ac40 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 1 -- 172.21.15.155:0/1081059649 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9634001c60 con 0x7f964009ac40 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f96458a8640 4 set_mon_vals no callback set 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.523+0000 7f963dffb640 10 monclient: dump: 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.517 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.524+0000 7f96458a8640 10 monclient: shutdown 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.524+0000 7f96458a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.524+0000 7f96458a8640 1 -- 172.21.15.155:0/1081059649 >> v1:172.21.15.155:6790/0 conn(0x7f964009ac40 legacy=0x7f96400b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.524+0000 7f96458a8640 1 -- 172.21.15.155:0/1081059649 shutdown_connections 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.524+0000 7f96458a8640 1 -- 172.21.15.155:0/1081059649 >> 172.21.15.155:0/1081059649 conn(0x7f9640066b00 msgr2=0x7f9640066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.524+0000 7f96458a8640 1 -- 172.21.15.155:0/1081059649 shutdown_connections 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.524+0000 7f96458a8640 1 -- 172.21.15.155:0/1081059649 wait complete. 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 10 monclient: build_initial_monmap 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 10 monclient: monmap: 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:55.519 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 1 librados: starting msgr at 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 1 librados: starting objecter 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 1 Processor -- start 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 1 -- start start 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 1 librados: setting wanted keys 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 1 librados: calling monclient init 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.525+0000 7f96458a8640 10 monclient: init 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:55.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient(hunting): picked mon.noname-b con 0x7f964009ac40 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9640139b60 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9640142a50 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96401ca0c0 con 0x7f9640142a50 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96401ca3a0 con 0x7f964009ac40 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96401c8840 con 0x7f9640139b60 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient(hunting): authenticate will time out at 818.817997s 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f963e7fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f9640139b60 0x7f96401ca7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37350/0 (socket says 172.21.15.155:37350) 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f963e7fc640 1 -- 172.21.15.155:0/3519917273 learned_addr learned my addr 172.21.15.155:0/3519917273 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 client.?.objecter ms_handle_connect 0x7f9640139b60 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 client.?.objecter ms_handle_connect 0x7f9640142a50 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 client.?.objecter ms_handle_connect 0x7f964009ac40 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108329972 0 0) 0x7f96401c8840 con 0x7f9640139b60 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962c004eb0 con 0x7f9640139b60 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932357598 0 0) 0x7f96401ca0c0 con 0x7f9640142a50 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96401c8840 con 0x7f9640142a50 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548541975 0 0) 0x7f96401ca3a0 con 0x7f964009ac40 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): my global_id is 4177 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96401ca0c0 con 0x7f964009ac40 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3754852262 0 0) 0x7f962c004eb0 con 0x7f9640139b60 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96401ca3a0 con 0x7f9640139b60 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3533298619 0 0) 0x7f96401c8840 con 0x7f9640142a50 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f962c004eb0 con 0x7f9640142a50 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1364320957 0 0) 0x7f96401ca0c0 con 0x7f964009ac40 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96401c8840 con 0x7f964009ac40 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9630001940 con 0x7f9640139b60 2024-03-29T10:51:55.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): dump: 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:55.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9638001b50 con 0x7f9640142a50 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): dump: 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9634001b60 con 0x7f964009ac40 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient(hunting): dump: 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4107062602 0 0) 0x7f96401ca3a0 con 0x7f9640139b60 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 >> v1:172.21.15.183:6789/0 conn(0x7f964009ac40 legacy=0x7f96401c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 >> v1:172.21.15.155:6789/0 conn(0x7f9640142a50 legacy=0x7f96401e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 monclient: found mon.c 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96401e61b0 con 0x7f9640139b60 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f9627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 5 monclient: authenticate success, global_id 4190 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient: _renew_subs 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96401e3180 con 0x7f9640139b60 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.526+0000 7f96458a8640 10 monclient: _renew_subs 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f96458a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96401e3660 con 0x7f9640139b60 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f96458a8640 1 librados: init done 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9630001d50 con 0x7f9640139b60 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9630004980 con 0x7f9640139b60 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f963d7fa640 4 set_mon_vals no callback set 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 10 monclient: dump: 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:55.528 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9630004e70 con 0x7f9640139b60 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f96300067d0 con 0x7f9640139b60 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 10 client.4190.objecter ms_dispatch 0x7f9640000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f9627fff640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T10:51:55.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f96458a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f96458a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f96458a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:55.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.527+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9620005080 con 0x7f9640139b60 2024-03-29T10:51:55.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.529+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9630006ab0 con 0x7f9640139b60 2024-03-29T10:51:55.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.529+0000 7f9627fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:55.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.529+0000 7f9627fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:55.530 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:55.534+0000 7f4900421040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:51:55.531 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.543+0000 7f5afda2d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:51:55.573 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:55.592+0000 7f4900421040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:51:55.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.606+0000 7f96458a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:55.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.606+0000 7f96458a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:55.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.606+0000 7f96458a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:51:55.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.606+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9620005330 con 0x7f9640139b60 2024-03-29T10:51:55.588 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.606+0000 7f9627fff640 1 -- 172.21.15.155:0/3519917273 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f9630006260 con 0x7f9640139b60 2024-03-29T10:51:55.588 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.606+0000 7f9627fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:55.588 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.606+0000 7f9627fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:55.589 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-29T10:51:55.589 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 10 librados: watch_flush enter 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 10 librados: watch_flush exit 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 10 client.4190.objecter successfully canceled tick 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 10 monclient: shutdown 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 >> v1:172.21.15.155:6790/0 conn(0x7f9640139b60 legacy=0x7f96401ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.607+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 shutdown_connections 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.608+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 >> 172.21.15.155:0/3519917273 conn(0x7f9640066b00 msgr2=0x7f964013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.608+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 shutdown_connections 2024-03-29T10:51:55.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.608+0000 7f96458a8640 1 -- 172.21.15.155:0/3519917273 wait complete. 2024-03-29T10:51:55.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-29T10:51:55.608+0000 7f96458a8640 1 librados: shutdown 2024-03-29T10:51:55.600 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:51:55.600 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:51:55.637 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.655+0000 7f5afda2d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:55.688 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:55.706+0000 7f4900421040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:55.733 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.751+0000 7f5afda2d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:51:55.783 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:55.802+0000 7f4900421040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:51:55.887 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.905+0000 7f5afda2d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:55.929 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:55.946+0000 7f5afda2d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:55.939 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:55.958+0000 7f4900421040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:51:55.948 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-03-29T10:51:55.967+0000 7fb3f5cf9640 -1 Falling back to public interface 2024-03-29T10:51:55.948 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-03-29T10:51:55.967+0000 7fddf0213640 -1 Falling back to public interface 2024-03-29T10:51:55.958 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-29T10:51:55.976+0000 7f7a6a237640 -1 Falling back to public interface 2024-03-29T10:51:55.961 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-29T10:51:55.979+0000 7f82e400d640 -1 Falling back to public interface 2024-03-29T10:51:55.963 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-03-29T10:51:55.982+0000 7f78b90c8640 -1 Falling back to public interface 2024-03-29T10:51:55.969 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-29T10:51:55.987+0000 7f2fb6973640 -1 Falling back to public interface 2024-03-29T10:51:55.970 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-29T10:51:55.987+0000 7f28816aa640 -1 Falling back to public interface 2024-03-29T10:51:55.970 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-29T10:51:55.987+0000 7f2880659640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2024-03-29T10:51:55.971 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-03-29T10:51:55.990+0000 7f5ddd215640 -1 Falling back to public interface 2024-03-29T10:51:55.987 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:56.005+0000 7f4900421040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:51:56.088 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:56.105+0000 7f5afda2d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:51:56.143 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:56.161+0000 7f4900421040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:51:56.407 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:56.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.514+0000 7f645c686640 10 monclient: get_monmap_and_config 2024-03-29T10:51:56.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.514+0000 7f645c686640 10 monclient: build_initial_monmap 2024-03-29T10:51:56.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.514+0000 7f645c686640 10 monclient: monmap: 2024-03-29T10:51:56.496 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:51:56.496 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:56.515608+0000 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:56.515608+0000 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:51:56.497 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.515+0000 7f645c686640 1 Processor -- start 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.515+0000 7f645c686640 1 -- start start 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.515+0000 7f645c686640 10 monclient: init 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 10 monclient(hunting): picked mon.noname-c con 0x7f64541457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 10 monclient(hunting): picked mon.noname-b con 0x7f645413f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 10 monclient(hunting): picked mon.noname-a con 0x7f64540a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:56.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64540aaef0 con 0x7f64540a9d70 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6454145bd0 con 0x7f64541457f0 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6454145eb0 con 0x7f645413f650 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645c686640 10 monclient(hunting): authenticate will time out at 819.808226s 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645ac23640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f64540a9d70 0x7f645414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50042/0 (socket says 172.21.15.155:50042) 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f645ac23640 1 -- 172.21.15.155:0/2602026174 learned_addr learned my addr 172.21.15.155:0/2602026174 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459c21640 1 --1- 172.21.15.155:0/2602026174 >> v1:172.21.15.155:6790/0 conn(0x7f645413f650 0x7f64540b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37362/0 (socket says 172.21.15.155:37362) 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366178942 0 0) 0x7f64540aaef0 con 0x7f64540a9d70 2024-03-29T10:51:56.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _negotiate 2024-03-29T10:51:56.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:56.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:56.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:51:56.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6440003520 con 0x7f64540a9d70 2024-03-29T10:51:56.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268513529 0 0) 0x7f6454145eb0 con 0x7f645413f650 2024-03-29T10:51:56.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _negotiate 2024-03-29T10:51:56.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:56.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:56.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:51:56.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64540aaef0 con 0x7f645413f650 2024-03-29T10:51:56.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029237698 0 0) 0x7f6454145bd0 con 0x7f64541457f0 2024-03-29T10:51:56.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _negotiate 2024-03-29T10:51:56.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:56.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 10 monclient(hunting): my global_id is 4180 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6454145eb0 con 0x7f64541457f0 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1030170552 0 0) 0x7f6440003520 con 0x7f64540a9d70 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.516+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 >> v1:172.21.15.155:6790/0 conn(0x7f645413f650 legacy=0x7f64540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 >> v1:172.21.15.183:6789/0 conn(0x7f64541457f0 legacy=0x7f645413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: _finish_hunting 0 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 1 monclient: found mon.noname-a 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:51:56.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454146340 con 0x7f64540a9d70 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: _finish_auth 0 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: _check_auth_tickets 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f644c001a10 con 0x7f64540a9d70 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: dump: 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:56.512 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 5 monclient: authenticate success, global_id 4194 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:56.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f644c001b70 con 0x7f64540a9d70 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 1 -- 172.21.15.155:0/2602026174 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f644c001f80 con 0x7f64540a9d70 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 4 set_mon_vals no callback set 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f6459420640 10 monclient: dump: 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.514 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 10 monclient: shutdown 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 1 -- 172.21.15.155:0/2602026174 >> v1:172.21.15.155:6789/0 conn(0x7f64540a9d70 legacy=0x7f645414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:56.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 1 -- 172.21.15.155:0/2602026174 shutdown_connections 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 1 -- 172.21.15.155:0/2602026174 >> 172.21.15.155:0/2602026174 conn(0x7f6454066a00 msgr2=0x7f6454066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 1 -- 172.21.15.155:0/2602026174 shutdown_connections 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.517+0000 7f645c686640 1 -- 172.21.15.155:0/2602026174 wait complete. 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 10 monclient: build_initial_monmap 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 10 monclient: monmap: 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:56.516 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 1 librados: starting msgr at 2024-03-29T10:51:56.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 1 librados: starting objecter 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 1 Processor -- start 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 1 -- start start 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 1 librados: setting wanted keys 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 1 librados: calling monclient init 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.518+0000 7f645c686640 10 monclient: init 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 10 monclient(hunting): picked mon.noname-b con 0x7f64540a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 10 monclient(hunting): picked mon.noname-a con 0x7f645413f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:51:56.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 10 monclient(hunting): picked mon.noname-c con 0x7f64541457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645413ad20 con 0x7f645413f650 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6454140cc0 con 0x7f64540a9d70 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645413af90 con 0x7f64541457f0 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f645c686640 10 monclient(hunting): authenticate will time out at 819.811253s 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f6459c21640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f645413f650 0x7f6454141110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50052/0 (socket says 172.21.15.155:50052) 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f6459c21640 1 -- 172.21.15.155:0/592581661 learned_addr learned my addr 172.21.15.155:0/592581661 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 client.?.objecter ms_handle_connect 0x7f645413f650 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:56.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 client.?.objecter ms_handle_connect 0x7f64541457f0 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 client.?.objecter ms_handle_connect 0x7f64540a9d70 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848234388 0 0) 0x7f645413af90 con 0x7f64541457f0 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): my global_id is 4196 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438004eb0 con 0x7f64541457f0 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691104708 0 0) 0x7f645413ad20 con 0x7f645413f650 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:56.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645413af90 con 0x7f645413f650 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303833059 0 0) 0x7f6454140cc0 con 0x7f64540a9d70 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:51:56.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645413ad20 con 0x7f64540a9d70 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135682896 0 0) 0x7f645413ad20 con 0x7f64540a9d70 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6454140cc0 con 0x7f64540a9d70 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6444001630 con 0x7f64540a9d70 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): dump: 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.522 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.523 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:56.523 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.523 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:56.523 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:56.523 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:56.523 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.519+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1759634466 0 0) 0x7f6438004eb0 con 0x7f64541457f0 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f645413ad20 con 0x7f64541457f0 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616408452 0 0) 0x7f645413af90 con 0x7f645413f650 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6438004eb0 con 0x7f645413f650 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f644c001e80 con 0x7f64541457f0 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient(hunting): dump: 2024-03-29T10:51:56.524 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6450001970 con 0x7f645413f650 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:56.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient(hunting): dump: 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:56.526 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3990517721 0 0) 0x7f6454140cc0 con 0x7f64540a9d70 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 >> v1:172.21.15.155:6790/0 conn(0x7f64541457f0 legacy=0x7f64541e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 >> v1:172.21.15.155:6789/0 conn(0x7f645413f650 legacy=0x7f6454141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 monclient: found mon.b 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645413b270 con 0x7f64540a9d70 2024-03-29T10:51:56.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: _finish_auth 0 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 5 monclient: authenticate success, global_id 4183 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 10 monclient: _renew_subs 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6454138f30 con 0x7f64540a9d70 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 10 monclient: _renew_subs 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6454139e50 con 0x7f64540a9d70 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f645c686640 1 librados: init done 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6444001ba0 con 0x7f64540a9d70 2024-03-29T10:51:56.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f64440037b0 con 0x7f64540a9d70 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f6458c1f640 4 set_mon_vals no callback set 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 monclient: dump: 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:51:56.529 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6444003ca0 con 0x7f64540a9d70 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f6444004620 con 0x7f64540a9d70 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 10 client.4183.objecter ms_dispatch 0x7f6454000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.520+0000 7f644affd640 20 client.4183.objecter dump_active .. 0 homeless 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.521+0000 7f645c686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:56.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.521+0000 7f645c686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:56.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.521+0000 7f645c686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:56.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.521+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c005080 con 0x7f64540a9d70 2024-03-29T10:51:56.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.523+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6444004920 con 0x7f64540a9d70 2024-03-29T10:51:56.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.523+0000 7f644affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:56.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.523+0000 7f644affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:56.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.583+0000 7f645c686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.583+0000 7f645c686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.583+0000 7f645c686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.583+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f643c005330 con 0x7f64540a9d70 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.584+0000 7f644affd640 1 -- 172.21.15.155:0/592581661 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f6444004060 con 0x7f64540a9d70 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.584+0000 7f644affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.584+0000 7f644affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:51:56.568 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:56.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 10 librados: watch_flush enter 2024-03-29T10:51:56.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 10 librados: watch_flush exit 2024-03-29T10:51:56.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 20 client.4183.objecter shutdown clearing up homeless session... 2024-03-29T10:51:56.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 10 client.4183.objecter successfully canceled tick 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 10 monclient: shutdown 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 >> v1:172.21.15.183:6789/0 conn(0x7f64540a9d70 legacy=0x7f64541405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 shutdown_connections 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 >> 172.21.15.155:0/592581661 conn(0x7f6454066a00 msgr2=0x7f64540b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 shutdown_connections 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 1 -- 172.21.15.155:0/592581661 wait complete. 2024-03-29T10:51:56.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:51:56.585+0000 7f645c686640 1 librados: shutdown 2024-03-29T10:51:56.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:56.898 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:56.915+0000 7f5afda2d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:51:56.978 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:56.997+0000 7f4900421040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:51:57.175 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:57.189+0000 7f5afda2d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:51:57.223 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:57.240+0000 7f5afda2d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:51:57.257 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:57.276+0000 7f4900421040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:51:57.310 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:57.328+0000 7f4900421040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:51:57.339 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:57.356+0000 7f5afda2d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:51:57.434 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-03-29T10:51:57.453+0000 7f4900421040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:51:58.352 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:58.370+0000 7f5a9de7d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:51:58.352 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-29T10:51:58.370+0000 7f5a9de7d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:51:59.434 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-03-29T10:51:59.453+0000 7f78b90c8640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:51:59.455 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-29T10:51:59.473+0000 7f7a6a237640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:51:59.482 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-03-29T10:51:59.500+0000 7fddf0213640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:51:59.489 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-03-29T10:51:59.508+0000 7fb3f5cf9640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:51:59.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-03-29T10:51:59.519+0000 7f5ddd215640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:51:59.507 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-29T10:51:59.525+0000 7f2fb6973640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:51:59.511 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-29T10:51:59.529+0000 7f28816aa640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:51:59.518 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-29T10:51:59.536+0000 7f82e400d640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:52:03.383 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.546+0000 7f8e6576e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.546+0000 7f8e6576e640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.546+0000 7f8e6576e640 10 monclient: monmap: 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:03.547467+0000 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:03.547467+0000 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:03.528 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:03.529 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:03.529 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.548+0000 7f8e6576e640 1 Processor -- start 2024-03-29T10:52:03.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.548+0000 7f8e6576e640 1 -- start start 2024-03-29T10:52:03.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.548+0000 7f8e6576e640 10 monclient: init 2024-03-29T10:52:03.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:03.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e58038c60 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:03.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e580324b0 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:03.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e580d11e0 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:03.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580caf80 con 0x7f8e580324b0 2024-03-29T10:52:03.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e58039850 con 0x7f8e580d11e0 2024-03-29T10:52:03.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580c78d0 con 0x7f8e58038c60 2024-03-29T10:52:03.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e6576e640 10 monclient(hunting): authenticate will time out at 826.841421s 2024-03-29T10:52:03.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e64f6d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f8e580d11e0 0x7f8e580d35d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:54888/0 (socket says 172.21.15.155:54888) 2024-03-29T10:52:03.534 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.549+0000 7f8e64f6d640 1 -- 172.21.15.155:0/1298681665 learned_addr learned my addr 172.21.15.155:0/1298681665 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:03.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571425366 0 0) 0x7f8e580caf80 con 0x7f8e580324b0 2024-03-29T10:52:03.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.543 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e50003520 con 0x7f8e580324b0 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114388828 0 0) 0x7f8e58039850 con 0x7f8e580d11e0 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): my global_id is 4222 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e580caf80 con 0x7f8e580d11e0 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001871796 0 0) 0x7f8e580c78d0 con 0x7f8e58038c60 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e58039850 con 0x7f8e58038c60 2024-03-29T10:52:03.544 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 161761438 0 0) 0x7f8e580caf80 con 0x7f8e580d11e0 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 >> v1:172.21.15.155:6790/0 conn(0x7f8e58038c60 legacy=0x7f8e580c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 >> v1:172.21.15.155:6789/0 conn(0x7f8e580324b0 legacy=0x7f8e580cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 monclient: found mon.noname-c 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580c7a60 con 0x7f8e580d11e0 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8e6004f730 con 0x7f8e580d11e0 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.545 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.550+0000 7f8e5effd640 10 monclient: dump: 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e6576e640 5 monclient: authenticate success, global_id 4222 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e6576e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:03.546 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e600008d0 con 0x7f8e580d11e0 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e5effd640 1 -- 172.21.15.155:0/1298681665 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8e600758b0 con 0x7f8e580d11e0 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e5effd640 10 monclient: dump: 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:03.547 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e6576e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.551+0000 7f8e6576e640 4 set_mon_vals no callback set 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.552+0000 7f8e6576e640 10 monclient: shutdown 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.552+0000 7f8e6576e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.552+0000 7f8e6576e640 1 -- 172.21.15.155:0/1298681665 >> v1:172.21.15.183:6789/0 conn(0x7f8e580d11e0 legacy=0x7f8e580d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.552+0000 7f8e6576e640 1 -- 172.21.15.155:0/1298681665 shutdown_connections 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.552+0000 7f8e6576e640 1 -- 172.21.15.155:0/1298681665 >> 172.21.15.155:0/1298681665 conn(0x7f8e58005c20 msgr2=0x7f8e58006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.552+0000 7f8e6576e640 1 -- 172.21.15.155:0/1298681665 shutdown_connections 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.552+0000 7f8e6576e640 1 -- 172.21.15.155:0/1298681665 wait complete. 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.553+0000 7f8e6576e640 10 monclient: build_initial_monmap 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.553+0000 7f8e6576e640 10 monclient: monmap: 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:03.548 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.554+0000 7f8e6576e640 1 librados: starting msgr at 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.554+0000 7f8e6576e640 1 librados: starting objecter 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.554+0000 7f8e6576e640 1 Processor -- start 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.554+0000 7f8e6576e640 1 -- start start 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.554+0000 7f8e6576e640 1 librados: setting wanted keys 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.554+0000 7f8e6576e640 1 librados: calling monclient init 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.554+0000 7f8e6576e640 10 monclient: init 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:03.549 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e580324b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e58038c60 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e580d11e0 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e58034e90 con 0x7f8e58038c60 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5815ea60 con 0x7f8e580d11e0 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e58161540 con 0x7f8e580324b0 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.555+0000 7f8e6576e640 10 monclient(hunting): authenticate will time out at 826.847870s 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e64f6d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f8e580d11e0 0x7f8e58160e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:55036/0 (socket says 172.21.15.155:55036) 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e64f6d640 1 -- 172.21.15.155:0/2740296581 learned_addr learned my addr 172.21.15.155:0/2740296581 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e5cff9640 10 client.?.objecter ms_handle_connect 0x7f8e580324b0 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e5cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.550 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e5cff9640 10 client.?.objecter ms_handle_connect 0x7f8e58038c60 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e5cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e5cff9640 10 client.?.objecter ms_handle_connect 0x7f8e580d11e0 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.556+0000 7f8e5cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633171576 0 0) 0x7f8e58161540 con 0x7f8e580324b0 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e44004eb0 con 0x7f8e580324b0 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258142406 0 0) 0x7f8e5815ea60 con 0x7f8e580d11e0 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): my global_id is 4225 2024-03-29T10:52:03.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e58161540 con 0x7f8e580d11e0 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953031721 0 0) 0x7f8e58034e90 con 0x7f8e58038c60 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.557+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5815ea60 con 0x7f8e58038c60 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 66458168 0 0) 0x7f8e44004eb0 con 0x7f8e580324b0 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e58034e90 con 0x7f8e580324b0 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8e4c001930 con 0x7f8e580324b0 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): dump: 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:03.552 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819550887 0 0) 0x7f8e58161540 con 0x7f8e580d11e0 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e44004eb0 con 0x7f8e580d11e0 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4290141716 0 0) 0x7f8e5815ea60 con 0x7f8e58038c60 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e58161540 con 0x7f8e58038c60 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8e6004f400 con 0x7f8e580d11e0 2024-03-29T10:52:03.553 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): dump: 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8e540018d0 con 0x7f8e58038c60 2024-03-29T10:52:03.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 10 monclient(hunting): dump: 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 212276247 0 0) 0x7f8e58034e90 con 0x7f8e580324b0 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.558+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 >> v1:172.21.15.183:6789/0 conn(0x7f8e580d11e0 legacy=0x7f8e58160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 >> v1:172.21.15.155:6789/0 conn(0x7f8e58038c60 legacy=0x7f8e5815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 1 monclient: found mon.c 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580c3d80 con 0x7f8e580324b0 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 5 monclient: authenticate success, global_id 4244 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 10 monclient: _renew_subs 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e580c50e0 con 0x7f8e580324b0 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:03.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 10 monclient: _renew_subs 2024-03-29T10:52:03.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e580c55e0 con 0x7f8e580324b0 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.559+0000 7f8e6576e640 1 librados: init done 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.561+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e4c003610 con 0x7f8e580324b0 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.561+0000 7f8e5cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.561+0000 7f8e5dffb640 4 set_mon_vals no callback set 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8e4c003920 con 0x7f8e580324b0 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e5cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e5cff9640 10 monclient: dump: 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:03.559 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e6576e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e6576e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e6576e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.562+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e40005080 con 0x7f8e580324b0 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.563+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f8e4c0198f0 con 0x7f8e580324b0 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.564+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f8e4c0837c0 con 0x7f8e580324b0 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.564+0000 7f8e5cff9640 10 client.4244.objecter ms_dispatch 0x7f8e60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:03.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.564+0000 7f8e5cff9640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:03.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.564+0000 7f8e5cff9640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:03.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.564+0000 7f8e5cff9640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T10:52:03.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.564+0000 7f8e5f7fe640 10 monclient: get_auth_request con 0x7f8e4406e060 auth_method 0 2024-03-29T10:52:03.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.565+0000 7f8e5cff9640 10 client.4244.objecter ms_handle_connect 0x7f8e4406e060 2024-03-29T10:52:03.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.572+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e4c04f100 con 0x7f8e580324b0 2024-03-29T10:52:03.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.572+0000 7f8e5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:03.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.572+0000 7f8e5cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:03.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.995+0000 7f8e6576e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:03.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.995+0000 7f8e6576e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:03.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.995+0000 7f8e6576e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:03.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.995+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e40005330 con 0x7f8e580324b0 2024-03-29T10:52:03.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.996+0000 7f8e5cff9640 1 -- 172.21.15.155:0/2740296581 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f8e4c082e70 con 0x7f8e580324b0 2024-03-29T10:52:03.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.996+0000 7f8e5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:03.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:03.996+0000 7f8e5cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:03.978 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:03.978 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 10 librados: watch_flush enter 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 10 librados: watch_flush exit 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 10 client.4244.objecter successfully canceled tick 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 >> v1:172.21.15.183:6816/1412921548 conn(0x7f8e4406e060 legacy=0x7f8e44070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 10 monclient: shutdown 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 >> v1:172.21.15.155:6790/0 conn(0x7f8e580324b0 legacy=0x7f8e580343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 shutdown_connections 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 >> 172.21.15.155:0/2740296581 conn(0x7f8e58005c20 msgr2=0x7f8e580043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:03.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 shutdown_connections 2024-03-29T10:52:03.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 1 -- 172.21.15.155:0/2740296581 wait complete. 2024-03-29T10:52:03.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:04.001+0000 7f8e6576e640 1 librados: shutdown 2024-03-29T10:52:03.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:10.804 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.973+0000 7f6c2c4db640 10 monclient: get_monmap_and_config 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.973+0000 7f6c2c4db640 10 monclient: build_initial_monmap 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.973+0000 7f6c2c4db640 10 monclient: monmap: 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:10.975601+0000 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:10.975601+0000 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:10.956 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:10.957 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:10.957 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:10.957 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:10.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.974+0000 7f6c2c4db640 1 Processor -- start 2024-03-29T10:52:10.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.974+0000 7f6c2c4db640 1 -- start start 2024-03-29T10:52:10.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.974+0000 7f6c2c4db640 10 monclient: init 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 10 monclient(hunting): picked mon.noname-a con 0x7f6c2406ee40 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2a277640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6c2406ee40 0x7f6c2413bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:38550/0 (socket says 172.21.15.155:38550) 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2a277640 1 -- 172.21.15.155:0/1446087995 learned_addr learned my addr 172.21.15.155:0/1446087995 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 10 monclient(hunting): picked mon.noname-c con 0x7f6c240aeda0 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 10 monclient(hunting): picked mon.noname-b con 0x7f6c2413c470 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2406fdb0 con 0x7f6c2406ee40 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24071090 con 0x7f6c240aeda0 2024-03-29T10:52:10.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24071370 con 0x7f6c2413c470 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.976+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528355463 0 0) 0x7f6c2406fdb0 con 0x7f6c2406ee40 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c2c4db640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c2c4db640 10 monclient(hunting): authenticate will time out at 834.269944s 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): _negotiate 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c10003520 con 0x7f6c2406ee40 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737267909 0 0) 0x7f6c24071370 con 0x7f6c2413c470 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): _negotiate 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:10.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c2406fdb0 con 0x7f6c2413c470 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388525109 0 0) 0x7f6c10003520 con 0x7f6c2406ee40 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 >> v1:172.21.15.155:6790/0 conn(0x7f6c2413c470 legacy=0x7f6c2414a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 >> v1:172.21.15.183:6789/0 conn(0x7f6c240aeda0 legacy=0x7f6c24071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient: _finish_hunting 0 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 monclient: found mon.noname-a 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.977+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24070220 con 0x7f6c2406ee40 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: _finish_auth 0 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: _check_auth_tickets 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6c20001690 con 0x7f6c2406ee40 2024-03-29T10:52:10.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c2c4db640 5 monclient: authenticate success, global_id 4248 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c2c4db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: dump: 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:10.964 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c200018b0 con 0x7f6c2406ee40 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c2c4db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 1 -- 172.21.15.155:0/1446087995 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6c20001c40 con 0x7f6c2406ee40 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c2c4db640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c2c4db640 4 set_mon_vals no callback set 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.978+0000 7f6c29275640 10 monclient: dump: 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:10.965 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.979+0000 7f6c2c4db640 10 monclient: shutdown 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.979+0000 7f6c2c4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.979+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 >> v1:172.21.15.155:6789/0 conn(0x7f6c2406ee40 legacy=0x7f6c2413bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.979+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 shutdown_connections 2024-03-29T10:52:10.966 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.979+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 >> 172.21.15.155:0/1446087995 conn(0x7f6c24067470 msgr2=0x7f6c240b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:10.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.979+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 shutdown_connections 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.979+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1446087995 wait complete. 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.980+0000 7f6c2c4db640 10 monclient: build_initial_monmap 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.980+0000 7f6c2c4db640 10 monclient: monmap: 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:10.976 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.981+0000 7f6c2c4db640 1 librados: starting msgr at 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.981+0000 7f6c2c4db640 1 librados: starting objecter 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.981+0000 7f6c2c4db640 1 Processor -- start 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.982+0000 7f6c2c4db640 1 -- start start 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.982+0000 7f6c2c4db640 1 librados: setting wanted keys 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.982+0000 7f6c2c4db640 1 librados: calling monclient init 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.982+0000 7f6c2c4db640 10 monclient: init 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.983+0000 7f6c2c4db640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.983+0000 7f6c2c4db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.983+0000 7f6c2c4db640 10 monclient(hunting): picked mon.noname-a con 0x7f6c2406ee40 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.983+0000 7f6c2a277640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6c2406ee40 0x7f6c2413e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39832/0 (socket says 172.21.15.155:39832) 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.983+0000 7f6c2a277640 1 -- 172.21.15.155:0/1730592848 learned_addr learned my addr 172.21.15.155:0/1730592848 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.983+0000 7f6c2c4db640 10 monclient(hunting): picked mon.noname-c con 0x7f6c240aeda0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:10.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c2c4db640 10 monclient(hunting): picked mon.noname-b con 0x7f6c2413c470 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c1affd640 10 client.?.objecter ms_handle_connect 0x7f6c2406ee40 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24135b90 con 0x7f6c2406ee40 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24134f80 con 0x7f6c2413c470 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c241351f0 con 0x7f6c240aeda0 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c2c4db640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c2c4db640 10 monclient(hunting): authenticate will time out at 834.277038s 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c1affd640 10 client.?.objecter ms_handle_connect 0x7f6c240aeda0 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c1affd640 10 client.?.objecter ms_handle_connect 0x7f6c2413c470 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.984+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407898010 0 0) 0x7f6c24134f80 con 0x7f6c2413c470 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:10.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): my global_id is 4231 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c08004eb0 con 0x7f6c2413c470 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199120947 0 0) 0x7f6c241351f0 con 0x7f6c240aeda0 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.985+0000 7f6c1affd640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c24134f80 con 0x7f6c240aeda0 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092889608 0 0) 0x7f6c24135b90 con 0x7f6c2406ee40 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:10.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c241351f0 con 0x7f6c2406ee40 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1481142970 0 0) 0x7f6c24134f80 con 0x7f6c240aeda0 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c24135b90 con 0x7f6c240aeda0 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6c140018b0 con 0x7f6c240aeda0 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): dump: 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:10.980 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 379489215 0 0) 0x7f6c241351f0 con 0x7f6c2406ee40 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.986+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c24134f80 con 0x7f6c2406ee40 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6c200015d0 con 0x7f6c2406ee40 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 10 monclient(hunting): dump: 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:10.981 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 668136670 0 0) 0x7f6c24135b90 con 0x7f6c240aeda0 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 >> v1:172.21.15.183:6789/0 conn(0x7f6c2413c470 legacy=0x7f6c241dd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 >> v1:172.21.15.155:6789/0 conn(0x7f6c2406ee40 legacy=0x7f6c2413e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 1 monclient: found mon.c 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24136730 con 0x7f6c240aeda0 2024-03-29T10:52:10.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 10 monclient: _finish_auth 0 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 10 monclient: _renew_subs 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c240b3700 con 0x7f6c240aeda0 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 10 monclient: _renew_subs 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2413cb20 con 0x7f6c240aeda0 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.987+0000 7f6c2c4db640 1 librados: init done 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.988+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c140035f0 con 0x7f6c240aeda0 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.988+0000 7f6c1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.988+0000 7f6c1bfff640 4 set_mon_vals no callback set 2024-03-29T10:52:10.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c2c4db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6c14003900 con 0x7f6c240aeda0 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c1affd640 10 monclient: dump: 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:10.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c2c4db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c2c4db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.989+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0c005080 con 0x7f6c240aeda0 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.990+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f6c140198d0 con 0x7f6c240aeda0 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.991+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f6c140837f0 con 0x7f6c240aeda0 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.991+0000 7f6c1affd640 10 client.4250.objecter ms_dispatch 0x7f6c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.991+0000 7f6c1affd640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.991+0000 7f6c1affd640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.991+0000 7f6c1affd640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.992+0000 7f6c2a277640 10 monclient: get_auth_request con 0x7f6c0806dfb0 auth_method 0 2024-03-29T10:52:10.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:10.992+0000 7f6c1affd640 10 client.4250.objecter ms_handle_connect 0x7f6c0806dfb0 2024-03-29T10:52:10.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.001+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c1404f0d0 con 0x7f6c240aeda0 2024-03-29T10:52:10.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.001+0000 7f6c1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:10.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.001+0000 7f6c1affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:11.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.331+0000 7f6c2c4db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:11.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.331+0000 7f6c2c4db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:11.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.331+0000 7f6c2c4db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:52:11.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.331+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c0c005330 con 0x7f6c240aeda0 2024-03-29T10:52:11.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.332+0000 7f6c1affd640 1 -- 172.21.15.155:0/1730592848 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f6c14051000 con 0x7f6c240aeda0 2024-03-29T10:52:11.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.332+0000 7f6c1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:11.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.332+0000 7f6c1affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:11.315 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:11.315 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.335+0000 7f6c2c4db640 10 librados: watch_flush enter 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 10 librados: watch_flush exit 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 10 client.4250.objecter successfully canceled tick 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 >> v1:172.21.15.183:6816/1412921548 conn(0x7f6c0806dfb0 legacy=0x7f6c08070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 10 monclient: shutdown 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:11.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 >> v1:172.21.15.155:6790/0 conn(0x7f6c240aeda0 legacy=0x7f6c24136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:11.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 shutdown_connections 2024-03-29T10:52:11.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 >> 172.21.15.155:0/1730592848 conn(0x7f6c24067470 msgr2=0x7f6c24071790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:11.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 shutdown_connections 2024-03-29T10:52:11.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 1 -- 172.21.15.155:0/1730592848 wait complete. 2024-03-29T10:52:11.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:11.336+0000 7f6c2c4db640 1 librados: shutdown 2024-03-29T10:52:11.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:18.140 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.309+0000 7f628e9b5640 10 monclient: get_monmap_and_config 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.309+0000 7f628e9b5640 10 monclient: build_initial_monmap 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.310+0000 7f628e9b5640 10 monclient: monmap: 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:18.311690+0000 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:18.311690+0000 2024-03-29T10:52:18.292 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:18.293 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:18.293 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:18.293 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:18.293 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:18.293 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:18.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.311+0000 7f628e9b5640 1 Processor -- start 2024-03-29T10:52:18.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.311+0000 7f628e9b5640 1 -- start start 2024-03-29T10:52:18.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.311+0000 7f628e9b5640 10 monclient: init 2024-03-29T10:52:18.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:18.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 10 monclient(hunting): picked mon.noname-b con 0x7f62881422b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 10 monclient(hunting): picked mon.noname-a con 0x7f6288142690 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 10 monclient(hunting): picked mon.noname-c con 0x7f62880b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6288142c80 con 0x7f6288142690 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62880aae40 con 0x7f62880b17d0 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628813fbd0 con 0x7f62881422b0 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.312+0000 7f628e9b5640 10 monclient(hunting): authenticate will time out at 841.605044s 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6287fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f62881422b0 0x7f628813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42201/0 (socket says 172.21.15.155:42201) 2024-03-29T10:52:18.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6287fff640 1 -- 172.21.15.155:0/1868646137 learned_addr learned my addr 172.21.15.155:0/1868646137 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:18.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f628cf52640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f62880b17d0 0x7f62880b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:42196/0 (socket says 172.21.15.155:42196) 2024-03-29T10:52:18.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162518582 0 0) 0x7f6288142c80 con 0x7f6288142690 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6270003520 con 0x7f6288142690 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137742400 0 0) 0x7f628813fbd0 con 0x7f62881422b0 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.315+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6288142c80 con 0x7f62881422b0 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 975099578 0 0) 0x7f6270003520 con 0x7f6288142690 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 >> v1:172.21.15.155:6790/0 conn(0x7f62881422b0 legacy=0x7f628813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:18.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 >> v1:172.21.15.183:6789/0 conn(0x7f62880b17d0 legacy=0x7f62880b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 monclient: found mon.noname-a 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6288151940 con 0x7f6288142690 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6280001a10 con 0x7f6288142690 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: dump: 2024-03-29T10:52:18.302 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f628e9b5640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f628e9b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6280001c30 con 0x7f6288142690 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f628e9b5640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f628e9b5640 4 set_mon_vals no callback set 2024-03-29T10:52:18.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 1 -- 172.21.15.155:0/1868646137 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6280001f40 con 0x7f6288142690 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.316+0000 7f6286ffd640 10 monclient: dump: 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:18.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.317+0000 7f628e9b5640 10 monclient: shutdown 2024-03-29T10:52:18.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.317+0000 7f628e9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:18.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.317+0000 7f628e9b5640 1 -- 172.21.15.155:0/1868646137 >> v1:172.21.15.155:6789/0 conn(0x7f6288142690 legacy=0x7f62880b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:18.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.318+0000 7f628e9b5640 1 -- 172.21.15.155:0/1868646137 shutdown_connections 2024-03-29T10:52:18.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.318+0000 7f628e9b5640 1 -- 172.21.15.155:0/1868646137 >> 172.21.15.155:0/1868646137 conn(0x7f6288066a00 msgr2=0x7f6288066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.319+0000 7f628e9b5640 1 -- 172.21.15.155:0/1868646137 shutdown_connections 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.319+0000 7f628e9b5640 1 -- 172.21.15.155:0/1868646137 wait complete. 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 10 monclient: build_initial_monmap 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 10 monclient: monmap: 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:18.306 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 1 librados: starting msgr at 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 1 librados: starting objecter 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 1 Processor -- start 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 1 -- start start 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 1 librados: setting wanted keys 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 1 librados: calling monclient init 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.320+0000 7f628e9b5640 10 monclient: init 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:18.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 10 monclient(hunting): picked mon.noname-a con 0x7f62880b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 10 monclient(hunting): picked mon.noname-c con 0x7f62881422b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 10 monclient(hunting): picked mon.noname-b con 0x7f6288142690 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6288070270 con 0x7f62880b17d0 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628814e4f0 con 0x7f6288142690 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6288071d00 con 0x7f62881422b0 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.321+0000 7f628e9b5640 10 monclient(hunting): authenticate will time out at 841.614581s 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.322+0000 7f628cf52640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f6288142690 0x7f6288070da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:42292/0 (socket says 172.21.15.155:42292) 2024-03-29T10:52:18.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.322+0000 7f628cf52640 1 -- 172.21.15.155:0/4091339805 learned_addr learned my addr 172.21.15.155:0/4091339805 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:18.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 client.?.objecter ms_handle_connect 0x7f6288142690 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615699493 0 0) 0x7f628814e4f0 con 0x7f6288142690 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6274004eb0 con 0x7f6288142690 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 client.?.objecter ms_handle_connect 0x7f62881422b0 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.326+0000 7f6284ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655067088 0 0) 0x7f6274004eb0 con 0x7f6288142690 2024-03-29T10:52:18.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f628814e4f0 con 0x7f6288142690 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f627c0018f0 con 0x7f6288142690 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:18.315 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412892666 0 0) 0x7f6288071d00 con 0x7f62881422b0 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.327+0000 7f6284ff9640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6274004eb0 con 0x7f62881422b0 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1803334111 0 0) 0x7f628814e4f0 con 0x7f6288142690 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 >> v1:172.21.15.155:6790/0 conn(0x7f62881422b0 legacy=0x7f6288070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 >> v1:172.21.15.155:6789/0 conn(0x7f62880b17d0 legacy=0x7f62881deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 1 monclient: found mon.b 2024-03-29T10:52:18.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6288071fd0 con 0x7f6288142690 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f6284ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f628e9b5640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f628e9b5640 10 monclient: _renew_subs 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f628e9b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.328+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62880714b0 con 0x7f6288142690 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.329+0000 7f628e9b5640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.329+0000 7f628e9b5640 10 monclient: _renew_subs 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.329+0000 7f628e9b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.329+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f628806e2e0 con 0x7f6288142690 2024-03-29T10:52:18.317 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.329+0000 7f628e9b5640 1 librados: init done 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f628e9b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f628e9b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f628e9b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6288075f10 con 0x7f6288142690 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f627c001ac0 con 0x7f6288142690 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f6284ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f627c004a90 con 0x7f6288142690 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f6284ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f6284ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.330+0000 7f6284ff9640 10 monclient: dump: 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.318 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:18.319 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:18.319 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:18.319 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:18.319 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:18.319 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:18.319 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:18.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.331+0000 7f62867fc640 4 set_mon_vals no callback set 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.334+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f627c004bf0 con 0x7f6288142690 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.335+0000 7f6287fff640 10 monclient: get_auth_request con 0x7f627406dfc0 auth_method 0 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.335+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f627c084ab0 con 0x7f6288142690 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.335+0000 7f6284ff9640 10 client.4234.objecter ms_dispatch 0x7f6288000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.335+0000 7f6284ff9640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.335+0000 7f6284ff9640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.335+0000 7f6284ff9640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T10:52:18.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.336+0000 7f6284ff9640 10 client.4234.objecter ms_handle_connect 0x7f627406dfc0 2024-03-29T10:52:18.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.340+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f627c0501c0 con 0x7f6288142690 2024-03-29T10:52:18.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.340+0000 7f6284ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:18.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.340+0000 7f6284ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:18.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.666+0000 7f628e9b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:18.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.666+0000 7f628e9b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:18.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.666+0000 7f628e9b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:18.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.666+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62880b2270 con 0x7f6288142690 2024-03-29T10:52:18.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.669+0000 7f6284ff9640 1 -- 172.21.15.155:0/4091339805 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f627c0522c0 con 0x7f6288142690 2024-03-29T10:52:18.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.669+0000 7f6284ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:18.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.669+0000 7f6284ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:18.652 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:18.652 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:18.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 10 librados: watch_flush enter 2024-03-29T10:52:18.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 10 librados: watch_flush exit 2024-03-29T10:52:18.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T10:52:18.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 10 client.4234.objecter successfully canceled tick 2024-03-29T10:52:18.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 >> v1:172.21.15.183:6816/1412921548 conn(0x7f627406dfc0 legacy=0x7f6274070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:18.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 10 monclient: shutdown 2024-03-29T10:52:18.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:18.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 >> v1:172.21.15.183:6789/0 conn(0x7f6288142690 legacy=0x7f6288070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:18.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 shutdown_connections 2024-03-29T10:52:18.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.673+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 >> 172.21.15.155:0/4091339805 conn(0x7f6288066a00 msgr2=0x7f628813d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:18.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.675+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 shutdown_connections 2024-03-29T10:52:18.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.675+0000 7f628e9b5640 1 -- 172.21.15.155:0/4091339805 wait complete. 2024-03-29T10:52:18.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:18.675+0000 7f628e9b5640 1 librados: shutdown 2024-03-29T10:52:18.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:25.481 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:25.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.644+0000 7fbf29431640 10 monclient: get_monmap_and_config 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.644+0000 7fbf29431640 10 monclient: build_initial_monmap 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.645+0000 7fbf29431640 10 monclient: monmap: 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:25.646661+0000 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:25.646661+0000 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.646+0000 7fbf29431640 1 Processor -- start 2024-03-29T10:52:25.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.646+0000 7fbf29431640 1 -- start start 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.646+0000 7fbf29431640 10 monclient: init 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.647+0000 7fbf29431640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.647+0000 7fbf29431640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.647+0000 7fbf29431640 10 monclient(hunting): picked mon.noname-b con 0x7fbf240a9750 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.647+0000 7fbf29431640 10 monclient(hunting): picked mon.noname-c con 0x7fbf240ae930 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf237fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbf240ae930 0x7fbf240ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:38054/0 (socket says 172.21.15.155:38054) 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf29431640 10 monclient(hunting): picked mon.noname-a con 0x7fbf240acab0 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf29431640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240ad650 con 0x7fbf240acab0 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf29431640 1 -- 172.21.15.155:0/2702813252 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240aedc0 con 0x7fbf240ae930 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf237fe640 1 -- 172.21.15.155:0/2702813252 learned_addr learned my addr 172.21.15.155:0/2702813252 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf29431640 1 -- 172.21.15.155:0/2702813252 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240ab750 con 0x7fbf240a9750 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf29431640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.648+0000 7fbf29431640 10 monclient(hunting): authenticate will time out at 848.941112s 2024-03-29T10:52:25.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.650+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471873698 0 0) 0x7fbf240aedc0 con 0x7fbf240ae930 2024-03-29T10:52:25.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.650+0000 7fbf22ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:25.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.650+0000 7fbf22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:25.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.650+0000 7fbf22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:25.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.650+0000 7fbf22ffd640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:52:25.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.650+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c003520 con 0x7fbf240ae930 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.650+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351368681 0 0) 0x7fbf1c001490 con 0x7fbf240acab0 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf240aedc0 con 0x7fbf240acab0 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380550511 0 0) 0x7fbf240ab750 con 0x7fbf240a9750 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:52:25.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf1c001490 con 0x7fbf240a9750 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165394756 0 0) 0x7fbf0c003520 con 0x7fbf240ae930 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 >> v1:172.21.15.155:6790/0 conn(0x7fbf240a9750 legacy=0x7fbf240ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 >> v1:172.21.15.155:6789/0 conn(0x7fbf240acab0 legacy=0x7fbf240acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 monclient: found mon.noname-c 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240ab9c0 con 0x7fbf240ae930 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbf14001610 con 0x7fbf240ae930 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:25.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.655+0000 7fbf22ffd640 10 monclient: dump: 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.656+0000 7fbf29431640 5 monclient: authenticate success, global_id 4237 2024-03-29T10:52:25.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.656+0000 7fbf29431640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf14001770 con 0x7fbf240ae930 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf29431640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf29431640 4 set_mon_vals no callback set 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf22ffd640 1 -- 172.21.15.155:0/2702813252 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbf14001b80 con 0x7fbf240ae930 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.659+0000 7fbf22ffd640 10 monclient: dump: 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:25.646 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.660+0000 7fbf29431640 10 monclient: shutdown 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.660+0000 7fbf29431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.660+0000 7fbf29431640 1 -- 172.21.15.155:0/2702813252 >> v1:172.21.15.183:6789/0 conn(0x7fbf240ae930 legacy=0x7fbf240ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.660+0000 7fbf29431640 1 -- 172.21.15.155:0/2702813252 shutdown_connections 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.660+0000 7fbf29431640 1 -- 172.21.15.155:0/2702813252 >> 172.21.15.155:0/2702813252 conn(0x7fbf240673e0 msgr2=0x7fbf240b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:25.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.660+0000 7fbf29431640 1 -- 172.21.15.155:0/2702813252 shutdown_connections 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.660+0000 7fbf29431640 1 -- 172.21.15.155:0/2702813252 wait complete. 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.661+0000 7fbf29431640 10 monclient: build_initial_monmap 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.661+0000 7fbf29431640 10 monclient: monmap: 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:25.648 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.662+0000 7fbf29431640 1 librados: starting msgr at 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.662+0000 7fbf29431640 1 librados: starting objecter 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.662+0000 7fbf29431640 1 Processor -- start 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.662+0000 7fbf29431640 1 -- start start 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.662+0000 7fbf29431640 1 librados: setting wanted keys 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.662+0000 7fbf29431640 1 librados: calling monclient init 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.662+0000 7fbf29431640 10 monclient: init 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.663+0000 7fbf29431640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.663+0000 7fbf29431640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.663+0000 7fbf29431640 10 monclient(hunting): picked mon.noname-b con 0x7fbf240a9750 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:25.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.663+0000 7fbf29431640 10 monclient(hunting): picked mon.noname-a con 0x7fbf240acab0 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:25.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.664+0000 7fbf23fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbf240a9750 0x7fbf24084460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:41128/0 (socket says 172.21.15.155:41128) 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf23fff640 1 -- 172.21.15.155:0/1715424961 learned_addr learned my addr 172.21.15.155:0/1715424961 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf29431640 10 monclient(hunting): picked mon.noname-c con 0x7fbf240ae930 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2414a6f0 con 0x7fbf240acab0 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2414a920 con 0x7fbf240a9750 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf24148d00 con 0x7fbf240ae930 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf29431640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf29431640 10 monclient(hunting): authenticate will time out at 848.958270s 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 10 client.?.objecter ms_handle_connect 0x7fbf240a9750 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949695840 0 0) 0x7fbf2414a920 con 0x7fbf240a9750 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:25.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.665+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf10004eb0 con 0x7fbf240a9750 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.666+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4182211251 0 0) 0x7fbf10004eb0 con 0x7fbf240a9750 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.666+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf2414a920 con 0x7fbf240a9750 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.666+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbf180019f0 con 0x7fbf240a9750 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.666+0000 7fbf20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.666+0000 7fbf20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.666+0000 7fbf20ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:25.652 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:25.653 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:25.653 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:25.653 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:25.653 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:25.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.666+0000 7fbf20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 183556461 0 0) 0x7fbf2414a920 con 0x7fbf240a9750 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 >> v1:172.21.15.155:6790/0 conn(0x7fbf240ae930 legacy=0x7fbf241f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 >> v1:172.21.15.155:6789/0 conn(0x7fbf240acab0 legacy=0x7fbf241f4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 1 monclient: found mon.b 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf241490b0 con 0x7fbf240a9750 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:25.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 10 monclient: _renew_subs 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf241489f0 con 0x7fbf240a9750 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 10 monclient: _renew_subs 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf24084b70 con 0x7fbf240a9750 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf29431640 1 librados: init done 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf18001cd0 con 0x7fbf240a9750 2024-03-29T10:52:25.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbf18003a60 con 0x7fbf240a9750 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.670+0000 7fbf20ff9640 10 monclient: dump: 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:25.659 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.671+0000 7fbf227fc640 4 set_mon_vals no callback set 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.672+0000 7fbf29431640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.672+0000 7fbf29431640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.672+0000 7fbf29431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.672+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf04005080 con 0x7fbf240a9750 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.673+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fbf18003bc0 con 0x7fbf240a9750 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.674+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fbf18084af0 con 0x7fbf240a9750 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.674+0000 7fbf20ff9640 10 client.4240.objecter ms_dispatch 0x7fbf24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.674+0000 7fbf20ff9640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.674+0000 7fbf20ff9640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.674+0000 7fbf20ff9640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T10:52:25.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.675+0000 7fbf237fe640 10 monclient: get_auth_request con 0x7fbf1006dc90 auth_method 0 2024-03-29T10:52:25.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.675+0000 7fbf20ff9640 10 client.4240.objecter ms_handle_connect 0x7fbf1006dc90 2024-03-29T10:52:25.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.680+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf180503d0 con 0x7fbf240a9750 2024-03-29T10:52:25.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.681+0000 7fbf20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:25.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.681+0000 7fbf20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:25.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.996+0000 7fbf29431640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:25.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.996+0000 7fbf29431640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:25.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.996+0000 7fbf29431640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:52:25.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.996+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf04005330 con 0x7fbf240a9750 2024-03-29T10:52:25.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.997+0000 7fbf20ff9640 1 -- 172.21.15.155:0/1715424961 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fbf18052300 con 0x7fbf240a9750 2024-03-29T10:52:25.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.997+0000 7fbf20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:25.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:25.997+0000 7fbf20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:25.980 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:25.980 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:25.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.001+0000 7fbf29431640 10 librados: watch_flush enter 2024-03-29T10:52:25.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.001+0000 7fbf29431640 10 librados: watch_flush exit 2024-03-29T10:52:25.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.001+0000 7fbf29431640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T10:52:25.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.001+0000 7fbf29431640 10 client.4240.objecter successfully canceled tick 2024-03-29T10:52:25.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.003+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 >> v1:172.21.15.183:6816/1412921548 conn(0x7fbf1006dc90 legacy=0x7fbf10070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:25.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.003+0000 7fbf29431640 10 monclient: shutdown 2024-03-29T10:52:25.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.003+0000 7fbf29431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:25.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.003+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 >> v1:172.21.15.183:6789/0 conn(0x7fbf240a9750 legacy=0x7fbf24084460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:25.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.003+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 shutdown_connections 2024-03-29T10:52:25.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.003+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 >> 172.21.15.155:0/1715424961 conn(0x7fbf240673e0 msgr2=0x7fbf24069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:25.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.004+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 shutdown_connections 2024-03-29T10:52:25.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.004+0000 7fbf29431640 1 -- 172.21.15.155:0/1715424961 wait complete. 2024-03-29T10:52:25.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:26.004+0000 7fbf29431640 1 librados: shutdown 2024-03-29T10:52:26.004 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:32.810 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:32.961 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.980+0000 7f1b0a73b640 10 monclient: get_monmap_and_config 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.980+0000 7f1b0a73b640 10 monclient: build_initial_monmap 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.980+0000 7f1b0a73b640 10 monclient: monmap: 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:32.981549+0000 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:32.981549+0000 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:32.962 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:32.963 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.982+0000 7f1b0a73b640 1 Processor -- start 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.982+0000 7f1b0a73b640 1 -- start start 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.982+0000 7f1b0a73b640 10 monclient: init 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 10 monclient(hunting): picked mon.noname-a con 0x7f1afc036a60 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 10 monclient(hunting): picked mon.noname-b con 0x7f1afc0c5de0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 10 monclient(hunting): picked mon.noname-c con 0x7f1afc0c3070 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:32.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc02cc60 con 0x7f1afc036a60 2024-03-29T10:52:32.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc02e040 con 0x7f1afc0c3070 2024-03-29T10:52:32.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc02e320 con 0x7f1afc0c5de0 2024-03-29T10:52:32.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:32.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.983+0000 7f1b0a73b640 10 monclient(hunting): authenticate will time out at 856.275518s 2024-03-29T10:52:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1b08f38640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f1afc0c5de0 0x7f1afc0c2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56431/0 (socket says 172.21.15.155:56431) 2024-03-29T10:52:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1b08f38640 1 -- 172.21.15.155:0/2309874384 learned_addr learned my addr 172.21.15.155:0/2309874384 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465938573 0 0) 0x7f1afc02e320 con 0x7f1afc0c5de0 2024-03-29T10:52:32.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1afbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:32.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1afbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:32.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:32.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1afbfff640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:52:32.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.987+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0003520 con 0x7f1afc0c5de0 2024-03-29T10:52:32.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075875075 0 0) 0x7f1afc02e040 con 0x7f1afc0c3070 2024-03-29T10:52:32.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1afc02e320 con 0x7f1afc0c3070 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360222294 0 0) 0x7f1af0003520 con 0x7f1afc0c5de0 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 >> v1:172.21.15.183:6789/0 conn(0x7f1afc0c3070 legacy=0x7f1afc0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 >> v1:172.21.15.155:6789/0 conn(0x7f1afc036a60 legacy=0x7f1afc0c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 monclient: found mon.noname-b 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc02c830 con 0x7f1afc0c5de0 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:32.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1b0404f730 con 0x7f1afc0c5de0 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.991+0000 7f1afbfff640 10 monclient: dump: 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:32.974 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:32.975 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1b0a73b640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1b0a73b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b040008d0 con 0x7f1afc0c5de0 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1afbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1afbfff640 1 -- 172.21.15.155:0/2309874384 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1b040758b0 con 0x7f1afc0c5de0 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1b0a73b640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1b0a73b640 4 set_mon_vals no callback set 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1afbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1afbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:32.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.992+0000 7f1afbfff640 10 monclient: dump: 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.994+0000 7f1b0a73b640 10 monclient: shutdown 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.994+0000 7f1b0a73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.994+0000 7f1b0a73b640 1 -- 172.21.15.155:0/2309874384 >> v1:172.21.15.155:6790/0 conn(0x7f1afc0c5de0 legacy=0x7f1afc0c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:32.977 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.995+0000 7f1b0a73b640 1 -- 172.21.15.155:0/2309874384 shutdown_connections 2024-03-29T10:52:32.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.995+0000 7f1b0a73b640 1 -- 172.21.15.155:0/2309874384 >> 172.21.15.155:0/2309874384 conn(0x7f1afc005c00 msgr2=0x7f1afc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:32.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.997+0000 7f1b0a73b640 1 -- 172.21.15.155:0/2309874384 shutdown_connections 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.998+0000 7f1b0a73b640 1 -- 172.21.15.155:0/2309874384 wait complete. 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 10 monclient: build_initial_monmap 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 10 monclient: monmap: 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:32.981 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 1 librados: starting msgr at 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 1 librados: starting objecter 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 1 Processor -- start 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 1 -- start start 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 1 librados: setting wanted keys 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 1 librados: calling monclient init 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:32.999+0000 7f1b0a73b640 10 monclient: init 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:32.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 10 monclient(hunting): picked mon.noname-b con 0x7f1afc036a60 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 10 monclient(hunting): picked mon.noname-c con 0x7f1afc0c3070 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 10 monclient(hunting): picked mon.noname-a con 0x7f1afc0075a0 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc0c4ff0 con 0x7f1afc0075a0 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc008a00 con 0x7f1afc036a60 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc0c99f0 con 0x7f1afc0c3070 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:32.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.000+0000 7f1b0a73b640 10 monclient(hunting): authenticate will time out at 856.292605s 2024-03-29T10:52:32.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.007+0000 7f1b08f38640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f1afc0c3070 0x7f1afc0c45c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56513/0 (socket says 172.21.15.155:56513) 2024-03-29T10:52:32.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.007+0000 7f1b08f38640 1 -- 172.21.15.155:0/1901721190 learned_addr learned my addr 172.21.15.155:0/1901721190 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.008+0000 7f1af9ffb640 10 client.?.objecter ms_handle_connect 0x7f1afc036a60 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.008+0000 7f1af9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.009+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020917696 0 0) 0x7f1afc008a00 con 0x7f1afc036a60 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae8004eb0 con 0x7f1afc036a60 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 client.?.objecter ms_handle_connect 0x7f1afc0075a0 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:32.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1b09739640 1 -- 172.21.15.155:0/1901721190 >> v1:172.21.15.183:6789/0 conn(0x7f1afc036a60 legacy=0x7f1afc0c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1b09739640 1 -- 172.21.15.155:0/1901721190 >> v1:172.21.15.183:6789/0 conn(0x7f1afc036a60 legacy=0x7f1afc0c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1b09739640 1 --1- 172.21.15.155:0/1901721190 >> v1:172.21.15.183:6789/0 conn(0x7f1afc036a60 0x7f1afc0c8c60 :-1 s=OPENED pgs=91578 cs=1 l=1).handle_message read tag failed 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1b09739640 1 --1- 172.21.15.155:0/1901721190 >> v1:172.21.15.183:6789/0 conn(0x7f1afc036a60 0x7f1afc0c8c60 :-1 s=OPENED pgs=91578 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.183:6789/0 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004247950 0 0) 0x7f1afc0c4ff0 con 0x7f1afc0075a0 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:52:32.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1afc008a00 con 0x7f1afc0075a0 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 client.?.objecter ms_handle_connect 0x7f1afc0c3070 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930016433 0 0) 0x7f1afc008a00 con 0x7f1afc0075a0 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1afc0c4ff0 con 0x7f1afc0075a0 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840771824 0 0) 0x7f1afc0c99f0 con 0x7f1afc0c3070 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1afc008a00 con 0x7f1afc0c3070 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1b000019a0 con 0x7f1afc0075a0 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:32.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.010+0000 7f1af9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.011+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 981726956 0 0) 0x7f1afc0c4ff0 con 0x7f1afc0075a0 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.011+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 >> v1:172.21.15.155:6790/0 conn(0x7f1afc0c3070 legacy=0x7f1afc0c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:32.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 >> v1:172.21.15.183:6789/0 conn(0x7f1afc036a60 legacy=0x7f1afc0c8c60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 1 monclient: found mon.a 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc0c4cd0 con 0x7f1afc0075a0 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1af9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 10 monclient: _renew_subs 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc009190 con 0x7f1afc0075a0 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 10 monclient: _renew_subs 2024-03-29T10:52:33.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:33.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc186ae0 con 0x7f1afc0075a0 2024-03-29T10:52:33.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.012+0000 7f1b0a73b640 1 librados: init done 2024-03-29T10:52:33.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.014+0000 7f1b0a73b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:33.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.014+0000 7f1b0a73b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:33.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.015+0000 7f1b0a73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.015+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc03ab60 con 0x7f1afc0075a0 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.016+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b00001db0 con 0x7f1afc0075a0 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.016+0000 7f1af9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.016+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1b00003800 con 0x7f1afc0075a0 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.016+0000 7f1af9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.016+0000 7f1af9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.016+0000 7f1af9ffb640 10 monclient: dump: 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:33.002 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:33.003 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:33.003 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:33.003 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.017+0000 7f1afb7fe640 4 set_mon_vals no callback set 2024-03-29T10:52:33.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.020+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f1b00003960 con 0x7f1afc0075a0 2024-03-29T10:52:33.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.021+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f1b00083660 con 0x7f1afc0075a0 2024-03-29T10:52:33.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.021+0000 7f1af9ffb640 10 client.4263.objecter ms_dispatch 0x7f1b04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:33.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.021+0000 7f1af9ffb640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:33.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.021+0000 7f1af9ffb640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:33.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.021+0000 7f1af9ffb640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T10:52:33.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.024+0000 7f1b09739640 10 monclient: get_auth_request con 0x7f1ae806db50 auth_method 0 2024-03-29T10:52:33.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.024+0000 7f1af9ffb640 10 client.4263.objecter ms_handle_connect 0x7f1ae806db50 2024-03-29T10:52:33.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.025+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b0004ef40 con 0x7f1afc0075a0 2024-03-29T10:52:33.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.025+0000 7f1af9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:33.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.025+0000 7f1af9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:33.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.359+0000 7f1b0a73b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:33.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.359+0000 7f1b0a73b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:33.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.359+0000 7f1b0a73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:33.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.359+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1afc005120 con 0x7f1afc0075a0 2024-03-29T10:52:33.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.360+0000 7f1af9ffb640 1 -- 172.21.15.155:0/1901721190 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f1b00050e70 con 0x7f1afc0075a0 2024-03-29T10:52:33.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.360+0000 7f1af9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:33.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.360+0000 7f1af9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:33.344 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:33.344 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:33.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.366+0000 7f1b0a73b640 10 librados: watch_flush enter 2024-03-29T10:52:33.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.367+0000 7f1b0a73b640 10 librados: watch_flush exit 2024-03-29T10:52:33.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.367+0000 7f1b0a73b640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T10:52:33.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.367+0000 7f1b0a73b640 10 client.4263.objecter successfully canceled tick 2024-03-29T10:52:33.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.367+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 >> v1:172.21.15.183:6816/1412921548 conn(0x7f1ae806db50 legacy=0x7f1ae8070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:33.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.367+0000 7f1b0a73b640 10 monclient: shutdown 2024-03-29T10:52:33.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.367+0000 7f1b0a73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:33.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.367+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 >> v1:172.21.15.155:6789/0 conn(0x7f1afc0075a0 legacy=0x7f1afc1853c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:33.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.368+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 shutdown_connections 2024-03-29T10:52:33.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.368+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 >> 172.21.15.155:0/1901721190 conn(0x7f1afc005c00 msgr2=0x7f1afc0c0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:33.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.368+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 shutdown_connections 2024-03-29T10:52:33.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.368+0000 7f1b0a73b640 1 -- 172.21.15.155:0/1901721190 wait complete. 2024-03-29T10:52:33.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:33.368+0000 7f1b0a73b640 1 librados: shutdown 2024-03-29T10:52:33.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:40.174 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:40.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.350+0000 7f63004c5640 10 monclient: get_monmap_and_config 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.350+0000 7f63004c5640 10 monclient: build_initial_monmap 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.350+0000 7f63004c5640 10 monclient: monmap: 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:40.351535+0000 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:40.351535+0000 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:40.333 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.353+0000 7f63004c5640 1 Processor -- start 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.353+0000 7f63004c5640 1 -- start start 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.353+0000 7f63004c5640 10 monclient: init 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.354+0000 7f63004c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 10 monclient(hunting): picked mon.noname-b con 0x7f62f80a9ac0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 10 monclient(hunting): picked mon.noname-a con 0x7f62f813ee90 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 10 monclient(hunting): picked mon.noname-c con 0x7f62f80af900 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f80afd10 con 0x7f62f813ee90 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f80af6f0 con 0x7f62f80af900 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f80aac80 con 0x7f62f80a9ac0 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:40.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.355+0000 7f63004c5640 10 monclient(hunting): authenticate will time out at 863.646976s 2024-03-29T10:52:40.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.359+0000 7f62fea62640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f62f80af900 0x7f62f8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:56499/0 (socket says 172.21.15.155:56499) 2024-03-29T10:52:40.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.359+0000 7f62fea62640 1 -- 172.21.15.155:0/911021743 learned_addr learned my addr 172.21.15.155:0/911021743 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:40.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.361+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881506700 0 0) 0x7f62f80aac80 con 0x7f62f80a9ac0 2024-03-29T10:52:40.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.361+0000 7f62fd25f640 10 monclient(hunting): _negotiate 2024-03-29T10:52:40.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.361+0000 7f62fd25f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:40.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.361+0000 7f62fd25f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:40.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.361+0000 7f62fd25f640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:52:40.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.361+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e4003520 con 0x7f62f80a9ac0 2024-03-29T10:52:40.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 617563483 0 0) 0x7f62e4003520 con 0x7f62f80a9ac0 2024-03-29T10:52:40.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 >> v1:172.21.15.183:6789/0 conn(0x7f62f80af900 legacy=0x7f62f8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:40.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 >> v1:172.21.15.155:6789/0 conn(0x7f62f813ee90 legacy=0x7f62f813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:40.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: _finish_hunting 0 2024-03-29T10:52:40.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 monclient: found mon.noname-b 2024-03-29T10:52:40.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f80aaf60 con 0x7f62f80a9ac0 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: _finish_auth 0 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: _check_auth_tickets 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f62f0001a10 con 0x7f62f80a9ac0 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: dump: 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:40.347 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f0001be0 con 0x7f62f80a9ac0 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 1 -- 172.21.15.155:0/911021743 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f62f0001ff0 con 0x7f62f80a9ac0 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.364+0000 7f62fd25f640 10 monclient: dump: 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.348 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.365+0000 7f63004c5640 5 monclient: authenticate success, global_id 4268 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.365+0000 7f63004c5640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:40.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.365+0000 7f63004c5640 4 set_mon_vals no callback set 2024-03-29T10:52:40.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.365+0000 7f63004c5640 10 monclient: shutdown 2024-03-29T10:52:40.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.365+0000 7f63004c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:40.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.365+0000 7f63004c5640 1 -- 172.21.15.155:0/911021743 >> v1:172.21.15.155:6790/0 conn(0x7f62f80a9ac0 legacy=0x7f62f80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:40.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.366+0000 7f63004c5640 1 -- 172.21.15.155:0/911021743 shutdown_connections 2024-03-29T10:52:40.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.366+0000 7f63004c5640 1 -- 172.21.15.155:0/911021743 >> 172.21.15.155:0/911021743 conn(0x7f62f80b35e0 msgr2=0x7f62f80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:40.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.368+0000 7f63004c5640 1 -- 172.21.15.155:0/911021743 shutdown_connections 2024-03-29T10:52:40.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.369+0000 7f63004c5640 1 -- 172.21.15.155:0/911021743 wait complete. 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.369+0000 7f63004c5640 10 monclient: build_initial_monmap 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.369+0000 7f63004c5640 10 monclient: monmap: 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.354 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.370+0000 7f63004c5640 1 librados: starting msgr at 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.370+0000 7f63004c5640 1 librados: starting objecter 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.370+0000 7f63004c5640 1 Processor -- start 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.370+0000 7f63004c5640 1 -- start start 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.370+0000 7f63004c5640 1 librados: setting wanted keys 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.370+0000 7f63004c5640 1 librados: calling monclient init 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.370+0000 7f63004c5640 10 monclient: init 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:40.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 10 monclient(hunting): picked mon.noname-c con 0x7f62f80af900 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:40.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 10 monclient(hunting): picked mon.noname-a con 0x7f62f81e77b0 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:40.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 10 monclient(hunting): picked mon.noname-b con 0x7f62f81ec960 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:40.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f81349e0 con 0x7f62f81e77b0 2024-03-29T10:52:40.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f81e8300 con 0x7f62f81ec960 2024-03-29T10:52:40.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8141f50 con 0x7f62f80af900 2024-03-29T10:52:40.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:40.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.371+0000 7f63004c5640 10 monclient(hunting): authenticate will time out at 863.663189s 2024-03-29T10:52:40.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.376+0000 7f62fe261640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f62f80af900 0x7f62f81405b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42275/0 (socket says 172.21.15.155:42275) 2024-03-29T10:52:40.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.376+0000 7f62fe261640 1 -- 172.21.15.155:0/4011862701 learned_addr learned my addr 172.21.15.155:0/4011862701 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:40.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.376+0000 7f62eeffd640 10 client.?.objecter ms_handle_connect 0x7f62f80af900 2024-03-29T10:52:40.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.376+0000 7f62eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:40.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.377+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276754656 0 0) 0x7f62f8141f50 con 0x7f62f80af900 2024-03-29T10:52:40.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.377+0000 7f62eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:40.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.377+0000 7f62eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:40.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.377+0000 7f62eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:40.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.377+0000 7f62eeffd640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:52:40.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.377+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62dc004eb0 con 0x7f62f80af900 2024-03-29T10:52:40.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.378+0000 7f62eeffd640 10 client.?.objecter ms_handle_connect 0x7f62f81ec960 2024-03-29T10:52:40.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.378+0000 7f62eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.378+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595238230 0 0) 0x7f62f81e8300 con 0x7f62f81ec960 2024-03-29T10:52:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.378+0000 7f62eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.378+0000 7f62eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.378+0000 7f62eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:40.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.378+0000 7f62eeffd640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f8141f50 con 0x7f62f81ec960 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 client.?.objecter ms_handle_connect 0x7f62f81e77b0 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028824693 0 0) 0x7f62f81349e0 con 0x7f62f81e77b0 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f81e8300 con 0x7f62f81e77b0 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4080750987 0 0) 0x7f62f8141f50 con 0x7f62f81ec960 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f81349e0 con 0x7f62f81ec960 2024-03-29T10:52:40.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f62f4005e50 con 0x7f62f81ec960 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): dump: 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:40.363 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073868310 0 0) 0x7f62f81e8300 con 0x7f62f81e77b0 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f8141f50 con 0x7f62f81e77b0 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f62e8001790 con 0x7f62f81e77b0 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): dump: 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.364 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3126667633 0 0) 0x7f62dc004eb0 con 0x7f62f80af900 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62f81e8300 con 0x7f62f80af900 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f62f0001f30 con 0x7f62f80af900 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:40.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.379+0000 7f62eeffd640 10 monclient(hunting): dump: 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3792151321 0 0) 0x7f62f8141f50 con 0x7f62f81e77b0 2024-03-29T10:52:40.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 >> v1:172.21.15.155:6790/0 conn(0x7f62f80af900 legacy=0x7f62f81405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 >> v1:172.21.15.183:6789/0 conn(0x7f62f81ec960 legacy=0x7f62f81eed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 1 monclient: found mon.a 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f813fb60 con 0x7f62f81e77b0 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 10 monclient: _renew_subs 2024-03-29T10:52:40.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f8140fa0 con 0x7f62f81e77b0 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 10 monclient: _renew_subs 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f81f0650 con 0x7f62f81e77b0 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f63004c5640 1 librados: init done 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62e8001c80 con 0x7f62f81e77b0 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f62e8003790 con 0x7f62f81e77b0 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:40.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62eeffd640 10 monclient: dump: 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:40.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.380+0000 7f62fca5e640 4 set_mon_vals no callback set 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.381+0000 7f63004c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.382+0000 7f63004c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.382+0000 7f63004c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.382+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e0005080 con 0x7f62f81e77b0 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.383+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f62e80038f0 con 0x7f62f81e77b0 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.383+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f62e80835e0 con 0x7f62f81e77b0 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.383+0000 7f62eeffd640 10 client.4266.objecter ms_dispatch 0x7f62f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.383+0000 7f62eeffd640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.383+0000 7f62eeffd640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.383+0000 7f62eeffd640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.387+0000 7f62fe261640 10 monclient: get_auth_request con 0x7f62dc06e110 auth_method 0 2024-03-29T10:52:40.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.387+0000 7f62eeffd640 10 client.4266.objecter ms_handle_connect 0x7f62dc06e110 2024-03-29T10:52:40.375 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.395+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62e804ecf0 con 0x7f62f81e77b0 2024-03-29T10:52:40.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.395+0000 7f62eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:40.376 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.395+0000 7f62eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:40.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.722+0000 7f63004c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:40.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.722+0000 7f63004c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:40.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.722+0000 7f63004c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:40.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.722+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62e0005330 con 0x7f62f81e77b0 2024-03-29T10:52:40.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.723+0000 7f62eeffd640 1 -- 172.21.15.155:0/4011862701 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f62e8050df0 con 0x7f62f81e77b0 2024-03-29T10:52:40.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.723+0000 7f62eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:40.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.723+0000 7f62eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:40.705 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:40.705 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:40.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.727+0000 7f63004c5640 10 librados: watch_flush enter 2024-03-29T10:52:40.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.727+0000 7f63004c5640 10 librados: watch_flush exit 2024-03-29T10:52:40.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.727+0000 7f63004c5640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T10:52:40.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.727+0000 7f63004c5640 10 client.4266.objecter successfully canceled tick 2024-03-29T10:52:40.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.728+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 >> v1:172.21.15.183:6816/1412921548 conn(0x7f62dc06e110 legacy=0x7f62dc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:40.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.728+0000 7f63004c5640 10 monclient: shutdown 2024-03-29T10:52:40.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.728+0000 7f63004c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:40.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.728+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 >> v1:172.21.15.155:6789/0 conn(0x7f62f81e77b0 legacy=0x7f62f81e7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:40.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.729+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 shutdown_connections 2024-03-29T10:52:40.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.729+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 >> 172.21.15.155:0/4011862701 conn(0x7f62f80b35e0 msgr2=0x7f62f80aca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:40.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.729+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 shutdown_connections 2024-03-29T10:52:40.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.729+0000 7f63004c5640 1 -- 172.21.15.155:0/4011862701 wait complete. 2024-03-29T10:52:40.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:40.730+0000 7f63004c5640 1 librados: shutdown 2024-03-29T10:52:40.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:47.540 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:47.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.717+0000 7f28dea4e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:47.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.717+0000 7f28dea4e640 10 monclient: build_initial_monmap 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.717+0000 7f28dea4e640 10 monclient: monmap: 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:47.718608+0000 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:47.718608+0000 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:47.712 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:47.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.719+0000 7f28dea4e640 1 Processor -- start 2024-03-29T10:52:47.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.719+0000 7f28dea4e640 1 -- start start 2024-03-29T10:52:47.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.719+0000 7f28dea4e640 10 monclient: init 2024-03-29T10:52:47.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:47.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:47.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 10 monclient(hunting): picked mon.noname-a con 0x7f28d80b0860 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 10 monclient(hunting): picked mon.noname-c con 0x7f28d813ef30 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 10 monclient(hunting): picked mon.noname-b con 0x7f28d80af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80b0c40 con 0x7f28d80b0860 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aed10 con 0x7f28d813ef30 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80afd50 con 0x7f28d80af8e0 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.720+0000 7f28dea4e640 10 monclient(hunting): authenticate will time out at 871.012034s 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.723+0000 7f28d7fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f28d80b0860 0x7f28d813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:60307/0 (socket says 172.21.15.155:60307) 2024-03-29T10:52:47.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.723+0000 7f28d7fff640 1 -- 172.21.15.155:0/1517178600 learned_addr learned my addr 172.21.15.155:0/1517178600 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:47.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366915790 0 0) 0x7f28d80b0c40 con 0x7f28d80b0860 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c0003520 con 0x7f28d80b0860 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288842405 0 0) 0x7f28c0003520 con 0x7f28d80b0860 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.724+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 >> v1:172.21.15.155:6790/0 conn(0x7f28d80af8e0 legacy=0x7f28d8149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.726+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 >> v1:172.21.15.183:6789/0 conn(0x7f28d813ef30 legacy=0x7f28d80af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 1 monclient: found mon.noname-a 2024-03-29T10:52:47.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80ab1b0 con 0x7f28d80b0860 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28dea4e640 5 monclient: authenticate success, global_id 4269 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28dea4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f28c8001690 con 0x7f28d80b0860 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28d6ffd640 10 monclient: dump: 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:47.719 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.727+0000 7f28dea4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:47.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28c8001920 con 0x7f28d80b0860 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28d6ffd640 1 -- 172.21.15.155:0/1517178600 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f28c8001cb0 con 0x7f28d80b0860 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28d6ffd640 10 monclient: dump: 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:47.721 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28dea4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.729+0000 7f28dea4e640 4 set_mon_vals no callback set 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.730+0000 7f28dea4e640 10 monclient: shutdown 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.730+0000 7f28dea4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:47.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.730+0000 7f28dea4e640 1 -- 172.21.15.155:0/1517178600 >> v1:172.21.15.155:6789/0 conn(0x7f28d80b0860 legacy=0x7f28d813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:47.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.731+0000 7f28dea4e640 1 -- 172.21.15.155:0/1517178600 shutdown_connections 2024-03-29T10:52:47.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.731+0000 7f28dea4e640 1 -- 172.21.15.155:0/1517178600 >> 172.21.15.155:0/1517178600 conn(0x7f28d806b660 msgr2=0x7f28d806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:47.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.731+0000 7f28dea4e640 1 -- 172.21.15.155:0/1517178600 shutdown_connections 2024-03-29T10:52:47.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.731+0000 7f28dea4e640 1 -- 172.21.15.155:0/1517178600 wait complete. 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.732+0000 7f28dea4e640 10 monclient: build_initial_monmap 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.732+0000 7f28dea4e640 10 monclient: monmap: 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:47.727 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.732+0000 7f28dea4e640 1 librados: starting msgr at 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.732+0000 7f28dea4e640 1 librados: starting objecter 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.733+0000 7f28dea4e640 1 Processor -- start 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.733+0000 7f28dea4e640 1 -- start start 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.733+0000 7f28dea4e640 1 librados: setting wanted keys 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.733+0000 7f28dea4e640 1 librados: calling monclient init 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.733+0000 7f28dea4e640 10 monclient: init 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 10 monclient(hunting): picked mon.noname-b con 0x7f28d80af8e0 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80b0860 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 10 monclient(hunting): picked mon.noname-a con 0x7f28d813ef30 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80b2830 con 0x7f28d813ef30 2024-03-29T10:52:47.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d81cccd0 con 0x7f28d80af8e0 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d81cce60 con 0x7f28d80b0860 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.734+0000 7f28dea4e640 10 monclient(hunting): authenticate will time out at 871.026089s 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.738+0000 7f28dcfeb640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f28d813ef30 0x7f28d80b3640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:60421/0 (socket says 172.21.15.155:60421) 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.738+0000 7f28dcfeb640 1 -- 172.21.15.155:0/1173233900 learned_addr learned my addr 172.21.15.155:0/1173233900 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 10 client.?.objecter ms_handle_connect 0x7f28d813ef30 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366578066 0 0) 0x7f28d80b2830 con 0x7f28d813ef30 2024-03-29T10:52:47.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:47.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:47.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:47.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:52:47.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.739+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c4004eb0 con 0x7f28d813ef30 2024-03-29T10:52:47.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.742+0000 7f28d4ff9640 10 client.?.objecter ms_handle_connect 0x7f28d80af8e0 2024-03-29T10:52:47.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745998126 0 0) 0x7f28c4004eb0 con 0x7f28d813ef30 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28d80b2830 con 0x7f28d813ef30 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447400944 0 0) 0x7f28d81cccd0 con 0x7f28d80af8e0 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c4004eb0 con 0x7f28d80af8e0 2024-03-29T10:52:47.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f28d0004380 con 0x7f28d813ef30 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.743+0000 7f28d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.745 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:47.746 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:47.746 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:47.746 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:47.746 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:47.746 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:47.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:47.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2436843661 0 0) 0x7f28d80b2830 con 0x7f28d813ef30 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 >> v1:172.21.15.155:6790/0 conn(0x7f28d80b0860 legacy=0x7f28d80b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 >> v1:172.21.15.183:6789/0 conn(0x7f28d80af8e0 legacy=0x7f28d80b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 1 monclient: found mon.a 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.744+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d81ccff0 con 0x7f28d813ef30 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28d4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28d4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:47.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 5 monclient: authenticate success, global_id 4272 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 10 monclient: _renew_subs 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d8201000 con 0x7f28d813ef30 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 10 monclient: _renew_subs 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d8201370 con 0x7f28d813ef30 2024-03-29T10:52:47.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28dea4e640 1 librados: init done 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28d0001250 con 0x7f28d813ef30 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f28d00044e0 con 0x7f28d813ef30 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.745+0000 7f28d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.746+0000 7f28d5ffb640 4 set_mon_vals no callback set 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.748+0000 7f28d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.748+0000 7f28d4ff9640 10 monclient: dump: 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:47.749 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.748+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f28d00016e0 con 0x7f28d813ef30 2024-03-29T10:52:47.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.749+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f28d0084da0 con 0x7f28d813ef30 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.749+0000 7f28d4ff9640 10 client.4272.objecter ms_dispatch 0x7f28d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.749+0000 7f28d4ff9640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.749+0000 7f28d4ff9640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.749+0000 7f28d4ff9640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.749+0000 7f28d7fff640 10 monclient: get_auth_request con 0x7f28c406dd60 auth_method 0 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.751+0000 7f28dea4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.751+0000 7f28dea4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:47.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.751+0000 7f28dea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:47.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.751+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b8005080 con 0x7f28d813ef30 2024-03-29T10:52:47.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.752+0000 7f28d4ff9640 10 client.4272.objecter ms_handle_connect 0x7f28c406dd60 2024-03-29T10:52:47.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.763+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28d0050680 con 0x7f28d813ef30 2024-03-29T10:52:47.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.763+0000 7f28d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:47.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:47.763+0000 7f28d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:48.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.082+0000 7f28dea4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:48.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.082+0000 7f28dea4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:48.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.082+0000 7f28dea4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:48.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.082+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28b8005330 con 0x7f28d813ef30 2024-03-29T10:52:48.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.083+0000 7f28d4ff9640 1 -- 172.21.15.155:0/1173233900 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f28d00525b0 con 0x7f28d813ef30 2024-03-29T10:52:48.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.083+0000 7f28d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:48.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.083+0000 7f28d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:48.064 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:48.064 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:48.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.086+0000 7f28dea4e640 10 librados: watch_flush enter 2024-03-29T10:52:48.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.086+0000 7f28dea4e640 10 librados: watch_flush exit 2024-03-29T10:52:48.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.086+0000 7f28dea4e640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T10:52:48.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.086+0000 7f28dea4e640 10 client.4272.objecter successfully canceled tick 2024-03-29T10:52:48.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 >> v1:172.21.15.183:6816/1412921548 conn(0x7f28c406dd60 legacy=0x7f28c4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 10 monclient: shutdown 2024-03-29T10:52:48.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:48.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 >> v1:172.21.15.155:6789/0 conn(0x7f28d813ef30 legacy=0x7f28d80b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:48.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 shutdown_connections 2024-03-29T10:52:48.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 >> 172.21.15.155:0/1173233900 conn(0x7f28d806b660 msgr2=0x7f28d813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:48.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 shutdown_connections 2024-03-29T10:52:48.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.087+0000 7f28dea4e640 1 -- 172.21.15.155:0/1173233900 wait complete. 2024-03-29T10:52:48.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:48.090+0000 7f28dea4e640 1 librados: shutdown 2024-03-29T10:52:48.095 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:54.903 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.076+0000 7f5e9e6b5640 10 monclient: get_monmap_and_config 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.076+0000 7f5e9e6b5640 10 monclient: build_initial_monmap 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.076+0000 7f5e9e6b5640 10 monclient: monmap: 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:52:55.077558+0000 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:52:55.077558+0000 2024-03-29T10:52:55.058 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:55.059 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:55.059 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:55.059 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:52:55.059 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:52:55.059 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:55.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.077+0000 7f5e9e6b5640 1 Processor -- start 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.078+0000 7f5e9e6b5640 1 -- start start 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.078+0000 7f5e9e6b5640 10 monclient: init 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.078+0000 7f5e9e6b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.078+0000 7f5e9e6b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.079+0000 7f5e9e6b5640 10 monclient(hunting): picked mon.noname-c con 0x7f5e980b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e9e6b5640 10 monclient(hunting): picked mon.noname-b con 0x7f5e98071eb0 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e9e6b5640 10 monclient(hunting): picked mon.noname-a con 0x7f5e98143d10 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e9e6b5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9806ef80 con 0x7f5e98143d10 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e9e6b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9806f260 con 0x7f5e980b1f60 2024-03-29T10:52:55.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e9e6b5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980b40c0 con 0x7f5e98071eb0 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e9e6b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e9e6b5640 10 monclient(hunting): authenticate will time out at 878.374718s 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e977fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5e98071eb0 0x7f5e98146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40419/0 (socket says 172.21.15.155:40419) 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.082+0000 7f5e977fe640 1 -- 172.21.15.155:0/1388185565 learned_addr learned my addr 172.21.15.155:0/1388185565 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827949891 0 0) 0x7f5e980b40c0 con 0x7f5e98071eb0 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e80003520 con 0x7f5e98071eb0 2024-03-29T10:52:55.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4119624079 0 0) 0x7f5e80003520 con 0x7f5e98071eb0 2024-03-29T10:52:55.066 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.083+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 >> v1:172.21.15.183:6789/0 conn(0x7f5e980b1f60 legacy=0x7f5e98143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:55.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.086+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 >> v1:172.21.15.155:6789/0 conn(0x7f5e98143d10 legacy=0x7f5e98146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:55.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:55.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 1 monclient: found mon.noname-b 2024-03-29T10:52:55.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:55.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:52:55.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e98155d70 con 0x7f5e98071eb0 2024-03-29T10:52:55.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:55.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:55.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e8c001690 con 0x7f5e98071eb0 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: dump: 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.071 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e9e6b5640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e9e6b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e8c001860 con 0x7f5e98071eb0 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 1 -- 172.21.15.155:0/1388185565 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e8c001cb0 con 0x7f5e98071eb0 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:55.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e96ffd640 10 monclient: dump: 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e9e6b5640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.087+0000 7f5e9e6b5640 4 set_mon_vals no callback set 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.088+0000 7f5e9e6b5640 10 monclient: shutdown 2024-03-29T10:52:55.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.088+0000 7f5e9e6b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:55.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.088+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/1388185565 >> v1:172.21.15.155:6790/0 conn(0x7f5e98071eb0 legacy=0x7f5e98146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.088+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/1388185565 shutdown_connections 2024-03-29T10:52:55.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.088+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/1388185565 >> 172.21.15.155:0/1388185565 conn(0x7f5e98066a00 msgr2=0x7f5e98066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:55.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.090+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/1388185565 shutdown_connections 2024-03-29T10:52:55.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.090+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/1388185565 wait complete. 2024-03-29T10:52:55.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.091+0000 7f5e9e6b5640 10 monclient: build_initial_monmap 2024-03-29T10:52:55.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.091+0000 7f5e9e6b5640 10 monclient: monmap: 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.092+0000 7f5e9e6b5640 1 librados: starting msgr at 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.092+0000 7f5e9e6b5640 1 librados: starting objecter 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.092+0000 7f5e9e6b5640 1 Processor -- start 2024-03-29T10:52:55.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.093+0000 7f5e9e6b5640 1 -- start start 2024-03-29T10:52:55.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.093+0000 7f5e9e6b5640 1 librados: setting wanted keys 2024-03-29T10:52:55.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.093+0000 7f5e9e6b5640 1 librados: calling monclient init 2024-03-29T10:52:55.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.093+0000 7f5e9e6b5640 10 monclient: init 2024-03-29T10:52:55.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.094+0000 7f5e9e6b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:55.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.094+0000 7f5e9e6b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:55.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.096+0000 7f5e9e6b5640 10 monclient(hunting): picked mon.noname-a con 0x7f5e980b1f60 addr v1:172.21.15.155:6789/0 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e9e6b5640 10 monclient(hunting): picked mon.noname-b con 0x7f5e98143d10 addr v1:172.21.15.183:6789/0 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e9e6b5640 10 monclient(hunting): picked mon.noname-c con 0x7f5e9806f620 addr v1:172.21.15.155:6790/0 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e9e6b5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e981328d0 con 0x7f5e980b1f60 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e9e6b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980703a0 con 0x7f5e98143d10 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e9e6b5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e98070680 con 0x7f5e9806f620 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e9e6b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e9e6b5640 10 monclient(hunting): authenticate will time out at 878.390722s 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e97fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5e980b1f60 0x7f5e98071010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:51383/0 (socket says 172.21.15.155:51383) 2024-03-29T10:52:55.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.098+0000 7f5e97fff640 1 -- 172.21.15.155:0/609665988 learned_addr learned my addr 172.21.15.155:0/609665988 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 10 client.?.objecter ms_handle_connect 0x7f5e980b1f60 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704100139 0 0) 0x7f5e981328d0 con 0x7f5e980b1f60 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e84004eb0 con 0x7f5e980b1f60 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.099+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3591925330 0 0) 0x7f5e84004eb0 con 0x7f5e980b1f60 2024-03-29T10:52:55.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e981328d0 con 0x7f5e980b1f60 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e90001bf0 con 0x7f5e980b1f60 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.082 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.086 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.086 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:55.086 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:55.086 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:55.086 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:55.086 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:55.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:55.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4240564171 0 0) 0x7f5e981328d0 con 0x7f5e980b1f60 2024-03-29T10:52:55.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.100+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 >> v1:172.21.15.155:6790/0 conn(0x7f5e9806f620 legacy=0x7f5e981321c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:55.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 >> v1:172.21.15.183:6789/0 conn(0x7f5e98143d10 legacy=0x7f5e98131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:55.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 1 monclient: found mon.a 2024-03-29T10:52:55.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:55.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e98070960 con 0x7f5e980b1f60 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 10 client.?.objecter ms_handle_connect 0x7f5e98143d10 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.102+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e90001d50 con 0x7f5e980b1f60 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.103+0000 7f5e94ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.103+0000 7f5e967fc640 4 set_mon_vals no callback set 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.103+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e90004e60 con 0x7f5e980b1f60 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.103+0000 7f5e94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.103+0000 7f5e94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.103+0000 7f5e94ff9640 10 monclient: dump: 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:52:55.091 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 5 monclient: authenticate success, global_id 4275 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 10 monclient: _renew_subs 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e981544d0 con 0x7f5e980b1f60 2024-03-29T10:52:55.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 10 monclient: _renew_subs 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e98155220 con 0x7f5e980b1f60 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.104+0000 7f5e9e6b5640 1 librados: init done 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f5e9001ae30 con 0x7f5e980b1f60 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e9e6b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e9e6b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e9e6b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e98075f10 con 0x7f5e980b1f60 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f5e90084e70 con 0x7f5e980b1f60 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e94ff9640 10 client.4275.objecter ms_dispatch 0x7f5e98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e94ff9640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e94ff9640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.107+0000 7f5e94ff9640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-29T10:52:55.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.108+0000 7f5e977fe640 10 monclient: get_auth_request con 0x7f5e8406dc60 auth_method 0 2024-03-29T10:52:55.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.111+0000 7f5e94ff9640 10 client.4275.objecter ms_handle_connect 0x7f5e8406dc60 2024-03-29T10:52:55.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.116+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e90050580 con 0x7f5e980b1f60 2024-03-29T10:52:55.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.117+0000 7f5e94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:55.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.117+0000 7f5e94ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:55.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.449+0000 7f5e9e6b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:55.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.449+0000 7f5e9e6b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:55.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.449+0000 7f5e9e6b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:52:55.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.449+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e98154660 con 0x7f5e980b1f60 2024-03-29T10:52:55.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.453+0000 7f5e94ff9640 1 -- 172.21.15.155:0/609665988 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f5e90052680 con 0x7f5e980b1f60 2024-03-29T10:52:55.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.453+0000 7f5e94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:55.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.453+0000 7f5e94ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:55.437 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:52:55.437 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:55.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.456+0000 7f5e9e6b5640 10 librados: watch_flush enter 2024-03-29T10:52:55.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 10 librados: watch_flush exit 2024-03-29T10:52:55.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-29T10:52:55.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 10 client.4275.objecter successfully canceled tick 2024-03-29T10:52:55.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 >> v1:172.21.15.183:6816/1412921548 conn(0x7f5e8406dc60 legacy=0x7f5e84070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 10 monclient: shutdown 2024-03-29T10:52:55.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:55.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 >> v1:172.21.15.155:6789/0 conn(0x7f5e980b1f60 legacy=0x7f5e98071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:55.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 shutdown_connections 2024-03-29T10:52:55.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.457+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 >> 172.21.15.155:0/609665988 conn(0x7f5e98066a00 msgr2=0x7f5e980b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:55.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.458+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 shutdown_connections 2024-03-29T10:52:55.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.458+0000 7f5e9e6b5640 1 -- 172.21.15.155:0/609665988 wait complete. 2024-03-29T10:52:55.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:52:55.458+0000 7f5e9e6b5640 1 librados: shutdown 2024-03-29T10:52:55.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:02.266 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:02.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.468+0000 7f5113954640 10 monclient: get_monmap_and_config 2024-03-29T10:53:02.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.468+0000 7f5113954640 10 monclient: build_initial_monmap 2024-03-29T10:53:02.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.468+0000 7f5113954640 10 monclient: monmap: 2024-03-29T10:53:02.451 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:02.451 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:02.451 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:02.470301+0000 2024-03-29T10:53:02.451 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:02.470301+0000 2024-03-29T10:53:02.452 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:02.452 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:02.452 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:02.452 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:02.452 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:02.452 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:02.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.471+0000 7f5113954640 1 Processor -- start 2024-03-29T10:53:02.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.472+0000 7f5113954640 1 -- start start 2024-03-29T10:53:02.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.472+0000 7f5113954640 10 monclient: init 2024-03-29T10:53:02.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.472+0000 7f5113954640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:02.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:02.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 10 monclient(hunting): picked mon.noname-b con 0x7f510c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:02.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 10 monclient(hunting): picked mon.noname-c con 0x7f510c13f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:02.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 10 monclient(hunting): picked mon.noname-a con 0x7f510c0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:02.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c0aaef0 con 0x7f510c0a9d70 2024-03-29T10:53:02.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c145bd0 con 0x7f510c13f650 2024-03-29T10:53:02.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c145eb0 con 0x7f510c1457f0 2024-03-29T10:53:02.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:02.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.473+0000 7f5113954640 10 monclient(hunting): authenticate will time out at 885.766001s 2024-03-29T10:53:02.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.480+0000 7f5110eef640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f510c13f650 0x7f510c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:53479/0 (socket says 172.21.15.155:53479) 2024-03-29T10:53:02.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.480+0000 7f5110eef640 1 -- 172.21.15.155:0/2980936034 learned_addr learned my addr 172.21.15.155:0/2980936034 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:02.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870092168 0 0) 0x7f510c145bd0 con 0x7f510c13f650 2024-03-29T10:53:02.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:02.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:02.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:02.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:53:02.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f8003520 con 0x7f510c13f650 2024-03-29T10:53:02.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716203756 0 0) 0x7f510c0aaef0 con 0x7f510c0a9d70 2024-03-29T10:53:02.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:02.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:02.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:02.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:53:02.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f510c145bd0 con 0x7f510c0a9d70 2024-03-29T10:53:02.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3481889905 0 0) 0x7f50f8003520 con 0x7f510c13f650 2024-03-29T10:53:02.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 >> v1:172.21.15.155:6790/0 conn(0x7f510c1457f0 legacy=0x7f510c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:02.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 >> v1:172.21.15.155:6789/0 conn(0x7f510c0a9d70 legacy=0x7f510c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:02.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:02.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 monclient: found mon.noname-c 2024-03-29T10:53:02.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:02.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:53:02.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c146340 con 0x7f510c13f650 2024-03-29T10:53:02.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:02.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:02.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:02.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f50fc001690 con 0x7f510c13f650 2024-03-29T10:53:02.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:02.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:02.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.481+0000 7f5103fff640 10 monclient: dump: 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:02.479 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:02.480 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:02.480 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:02.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.482+0000 7f5113954640 5 monclient: authenticate success, global_id 4261 2024-03-29T10:53:02.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.482+0000 7f5113954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:02.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.482+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50fc0017f0 con 0x7f510c13f650 2024-03-29T10:53:02.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.482+0000 7f5103fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.482+0000 7f5103fff640 1 -- 172.21.15.155:0/2980936034 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f50fc001c00 con 0x7f510c13f650 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.482+0000 7f5113954640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.482+0000 7f5113954640 4 set_mon_vals no callback set 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.484+0000 7f5113954640 10 monclient: shutdown 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.484+0000 7f5113954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.484+0000 7f5113954640 1 -- 172.21.15.155:0/2980936034 >> v1:172.21.15.183:6789/0 conn(0x7f510c13f650 legacy=0x7f510c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.484+0000 7f5103fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.484+0000 7f5113954640 1 -- 172.21.15.155:0/2980936034 shutdown_connections 2024-03-29T10:53:02.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.484+0000 7f5113954640 1 -- 172.21.15.155:0/2980936034 >> 172.21.15.155:0/2980936034 conn(0x7f510c066a00 msgr2=0x7f510c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:02.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.486+0000 7f5113954640 1 -- 172.21.15.155:0/2980936034 shutdown_connections 2024-03-29T10:53:02.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.486+0000 7f5113954640 1 -- 172.21.15.155:0/2980936034 wait complete. 2024-03-29T10:53:02.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.488+0000 7f5113954640 10 monclient: build_initial_monmap 2024-03-29T10:53:02.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.488+0000 7f5113954640 10 monclient: monmap: 2024-03-29T10:53:02.482 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:02.483 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:02.483 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:02.483 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:02.483 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:02.483 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:02.483 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:02.483 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.488+0000 7f5113954640 1 librados: starting msgr at 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.488+0000 7f5113954640 1 librados: starting objecter 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.488+0000 7f5113954640 1 Processor -- start 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.489+0000 7f5113954640 1 -- start start 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.489+0000 7f5113954640 1 librados: setting wanted keys 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.489+0000 7f5113954640 1 librados: calling monclient init 2024-03-29T10:53:02.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.489+0000 7f5113954640 10 monclient: init 2024-03-29T10:53:02.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:02.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:02.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 10 monclient(hunting): picked mon.noname-b con 0x7f510c0a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:02.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 10 monclient(hunting): picked mon.noname-a con 0x7f510c1457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:02.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 10 monclient(hunting): picked mon.noname-c con 0x7f510c138cc0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:02.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c1da7f0 con 0x7f510c1457f0 2024-03-29T10:53:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c0ad350 con 0x7f510c0a9d70 2024-03-29T10:53:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c1db3e0 con 0x7f510c138cc0 2024-03-29T10:53:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:02.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.490+0000 7f5113954640 10 monclient(hunting): authenticate will time out at 885.783637s 2024-03-29T10:53:02.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.494+0000 7f51116f0640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f510c0a9d70 0x7f510c0abb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:53579/0 (socket says 172.21.15.155:53579) 2024-03-29T10:53:02.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.494+0000 7f51116f0640 1 -- 172.21.15.155:0/218390457 learned_addr learned my addr 172.21.15.155:0/218390457 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:02.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.494+0000 7f5101ffb640 10 client.?.objecter ms_handle_connect 0x7f510c0a9d70 2024-03-29T10:53:02.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.494+0000 7f5101ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.495+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788458661 0 0) 0x7f510c0ad350 con 0x7f510c0a9d70 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.495+0000 7f5101ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.495+0000 7f5101ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.495+0000 7f5101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.495+0000 7f5101ffb640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.495+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50f0004eb0 con 0x7f510c0a9d70 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.499+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2557295743 0 0) 0x7f50f0004eb0 con 0x7f510c0a9d70 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.499+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f510c0ad350 con 0x7f510c0a9d70 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.499+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f51080017d0 con 0x7f510c0a9d70 2024-03-29T10:53:02.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.499+0000 7f5101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.499+0000 7f5101ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.499+0000 7f5101ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:02.494 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.499+0000 7f5101ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1294762711 0 0) 0x7f510c0ad350 con 0x7f510c0a9d70 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 >> v1:172.21.15.155:6790/0 conn(0x7f510c138cc0 legacy=0x7f510c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 >> v1:172.21.15.155:6789/0 conn(0x7f510c1457f0 legacy=0x7f510c1dacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 1 monclient: found mon.b 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:02.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c1db570 con 0x7f510c0a9d70 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.500+0000 7f5101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5108001930 con 0x7f510c0a9d70 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5101ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5108004a00 con 0x7f510c0a9d70 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5101ffb640 10 monclient: dump: 2024-03-29T10:53:02.496 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5113954640 5 monclient: authenticate success, global_id 4264 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5113954640 10 monclient: _renew_subs 2024-03-29T10:53:02.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5113954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.501+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510c13b730 con 0x7f510c0a9d70 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.502+0000 7f5113954640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.502+0000 7f5113954640 10 monclient: _renew_subs 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.502+0000 7f5113954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.502+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f510c1383e0 con 0x7f510c0a9d70 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.502+0000 7f5113954640 1 librados: init done 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.502+0000 7f51037fe640 4 set_mon_vals no callback set 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.505+0000 7f5113954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.505+0000 7f5113954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.505+0000 7f5113954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.505+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510c1da650 con 0x7f510c0a9d70 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.507+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f510801a9d0 con 0x7f510c0a9d70 2024-03-29T10:53:02.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.507+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f5108084aa0 con 0x7f510c0a9d70 2024-03-29T10:53:02.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.507+0000 7f5101ffb640 10 client.4264.objecter ms_dispatch 0x7f510c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:02.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.507+0000 7f5101ffb640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:02.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.507+0000 7f5101ffb640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:02.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.507+0000 7f5101ffb640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T10:53:02.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.508+0000 7f5110eef640 10 monclient: get_auth_request con 0x7f50f006dcd0 auth_method 0 2024-03-29T10:53:02.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.508+0000 7f5101ffb640 10 client.4264.objecter ms_handle_connect 0x7f50f006dcd0 2024-03-29T10:53:02.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.515+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f510c1da650 con 0x7f510c0a9d70 2024-03-29T10:53:02.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.515+0000 7f5101ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:02.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.516+0000 7f5101ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.874+0000 7f5113954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.874+0000 7f5113954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.874+0000 7f5113954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.874+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f510c1da650 con 0x7f510c0a9d70 2024-03-29T10:53:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.875+0000 7f5101ffb640 1 -- 172.21.15.155:0/218390457 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f510c1da650 con 0x7f510c0a9d70 2024-03-29T10:53:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.875+0000 7f5101ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:02.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.875+0000 7f5101ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:02.861 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:02.861 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:02.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.879+0000 7f5113954640 10 librados: watch_flush enter 2024-03-29T10:53:02.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.879+0000 7f5113954640 10 librados: watch_flush exit 2024-03-29T10:53:02.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.879+0000 7f5113954640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T10:53:02.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.879+0000 7f5113954640 10 client.4264.objecter successfully canceled tick 2024-03-29T10:53:02.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.880+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 >> v1:172.21.15.183:6816/1412921548 conn(0x7f50f006dcd0 legacy=0x7f50f0070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.880+0000 7f5113954640 10 monclient: shutdown 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.880+0000 7f5113954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.880+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 >> v1:172.21.15.183:6789/0 conn(0x7f510c0a9d70 legacy=0x7f510c0abb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.881+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 shutdown_connections 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.881+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 >> 172.21.15.155:0/218390457 conn(0x7f510c066a00 msgr2=0x7f510c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.881+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 shutdown_connections 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.881+0000 7f5113954640 1 -- 172.21.15.155:0/218390457 wait complete. 2024-03-29T10:53:02.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:02.882+0000 7f5113954640 1 librados: shutdown 2024-03-29T10:53:02.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:09.692 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:09.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.872+0000 7f829f997640 10 monclient: get_monmap_and_config 2024-03-29T10:53:09.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.872+0000 7f829f997640 10 monclient: build_initial_monmap 2024-03-29T10:53:09.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.872+0000 7f829f997640 10 monclient: monmap: 2024-03-29T10:53:09.854 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:09.854 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:09.873946+0000 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:09.873946+0000 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:09.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.873+0000 7f829f997640 1 Processor -- start 2024-03-29T10:53:09.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.873+0000 7f829f997640 1 -- start start 2024-03-29T10:53:09.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.873+0000 7f829f997640 10 monclient: init 2024-03-29T10:53:09.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:09.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:09.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 10 monclient(hunting): picked mon.noname-c con 0x7f82980b0770 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:09.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 10 monclient(hunting): picked mon.noname-b con 0x7f829814a700 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:09.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 10 monclient(hunting): picked mon.noname-a con 0x7f82980ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:09.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82980aba70 con 0x7f82980ac5a0 2024-03-29T10:53:09.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82980aecb0 con 0x7f82980b0770 2024-03-29T10:53:09.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82980af150 con 0x7f829814a700 2024-03-29T10:53:09.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:09.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829f997640 10 monclient(hunting): authenticate will time out at 893.167458s 2024-03-29T10:53:09.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829cf32640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f829814a700 0x7f82980abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33490/0 (socket says 172.21.15.155:33490) 2024-03-29T10:53:09.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.874+0000 7f829cf32640 1 -- 172.21.15.155:0/1085205787 learned_addr learned my addr 172.21.15.155:0/1085205787 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458500061 0 0) 0x7f82980af150 con 0x7f829814a700 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8284003520 con 0x7f829814a700 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26557077 0 0) 0x7f82980aecb0 con 0x7f82980b0770 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:53:09.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82980af150 con 0x7f82980b0770 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.875+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1008808870 0 0) 0x7f82980af150 con 0x7f82980b0770 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 >> v1:172.21.15.155:6790/0 conn(0x7f829814a700 legacy=0x7f82980abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 >> v1:172.21.15.155:6789/0 conn(0x7f82980ac5a0 legacy=0x7f8298151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 1 monclient: found mon.noname-c 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82980ac9d0 con 0x7f82980b0770 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: _finish_auth 0 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f82940015b0 con 0x7f82980b0770 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f829f997640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f829f997640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:09.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: dump: 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f829f997640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8294001780 con 0x7f82980b0770 2024-03-29T10:53:09.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 1 -- 172.21.15.155:0/1085205787 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8294001b50 con 0x7f82980b0770 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f828ffff640 10 monclient: dump: 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:09.862 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:09.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f829f997640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:09.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.876+0000 7f829f997640 4 set_mon_vals no callback set 2024-03-29T10:53:09.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.877+0000 7f829f997640 10 monclient: shutdown 2024-03-29T10:53:09.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.877+0000 7f829f997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:09.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.877+0000 7f829f997640 1 -- 172.21.15.155:0/1085205787 >> v1:172.21.15.183:6789/0 conn(0x7f82980b0770 legacy=0x7f8298149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:09.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.877+0000 7f829f997640 1 -- 172.21.15.155:0/1085205787 shutdown_connections 2024-03-29T10:53:09.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.877+0000 7f829f997640 1 -- 172.21.15.155:0/1085205787 >> 172.21.15.155:0/1085205787 conn(0x7f829806b660 msgr2=0x7f829806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:09.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.878+0000 7f829f997640 1 -- 172.21.15.155:0/1085205787 shutdown_connections 2024-03-29T10:53:09.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.878+0000 7f829f997640 1 -- 172.21.15.155:0/1085205787 wait complete. 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.879+0000 7f829f997640 10 monclient: build_initial_monmap 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.879+0000 7f829f997640 10 monclient: monmap: 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.879+0000 7f829f997640 1 librados: starting msgr at 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.879+0000 7f829f997640 1 librados: starting objecter 2024-03-29T10:53:09.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.879+0000 7f829f997640 1 Processor -- start 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.880+0000 7f829f997640 1 -- start start 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.880+0000 7f829f997640 1 librados: setting wanted keys 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.880+0000 7f829f997640 1 librados: calling monclient init 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.880+0000 7f829f997640 10 monclient: init 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 10 monclient(hunting): picked mon.noname-c con 0x7f82980ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829d733640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f82980ac5a0 0x7f82981452f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33772/0 (socket says 172.21.15.155:33772) 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829d733640 1 -- 172.21.15.155:0/2258108620 learned_addr learned my addr 172.21.15.155:0/2258108620 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 10 monclient(hunting): picked mon.noname-a con 0x7f82980b0770 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f828dffb640 10 client.?.objecter ms_handle_connect 0x7f82980ac5a0 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f828dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f828dffb640 10 client.?.objecter ms_handle_connect 0x7f82980b0770 2024-03-29T10:53:09.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f828dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 10 monclient(hunting): picked mon.noname-b con 0x7f829814a700 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8298128770 con 0x7f82980b0770 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82981289e0 con 0x7f829814a700 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8298126740 con 0x7f82980ac5a0 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.881+0000 7f829f997640 10 monclient(hunting): authenticate will time out at 893.174727s 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998463403 0 0) 0x7f8298128770 con 0x7f82980b0770 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f827c004eb0 con 0x7f82980b0770 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 client.?.objecter ms_handle_connect 0x7f829814a700 2024-03-29T10:53:09.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146311409 0 0) 0x7f8298126740 con 0x7f82980ac5a0 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8298128770 con 0x7f82980ac5a0 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 970522774 0 0) 0x7f827c004eb0 con 0x7f82980b0770 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8298126740 con 0x7f82980b0770 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8290001b90 con 0x7f82980b0770 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): dump: 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:09.872 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.882+0000 7f828dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1819014467 0 0) 0x7f8298128770 con 0x7f82980ac5a0 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f827c004eb0 con 0x7f82980ac5a0 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8294001a90 con 0x7f82980ac5a0 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:09.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 10 monclient(hunting): dump: 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2565582325 0 0) 0x7f8298126740 con 0x7f82980b0770 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 >> v1:172.21.15.155:6790/0 conn(0x7f82980ac5a0 legacy=0x7f82981452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 >> v1:172.21.15.183:6789/0 conn(0x7f829814a700 legacy=0x7f8298127370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:09.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 monclient: found mon.a 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8298126380 con 0x7f82980b0770 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f828dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 5 monclient: authenticate success, global_id 4281 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 10 monclient: _renew_subs 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:09.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82981dfa40 con 0x7f82980b0770 2024-03-29T10:53:09.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:09.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 10 monclient: _renew_subs 2024-03-29T10:53:09.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:09.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82981df020 con 0x7f82980b0770 2024-03-29T10:53:09.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.883+0000 7f829f997640 1 librados: init done 2024-03-29T10:53:09.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.884+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8290001fe0 con 0x7f82980b0770 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.884+0000 7f828dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.884+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8290004d00 con 0x7f82980b0770 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.884+0000 7f828dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.884+0000 7f828dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.884+0000 7f828dffb640 10 monclient: dump: 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:09.877 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.884+0000 7f828f7fe640 4 set_mon_vals no callback set 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.885+0000 7f829f997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.885+0000 7f829f997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.885+0000 7f829f997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.885+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8278005080 con 0x7f82980b0770 2024-03-29T10:53:09.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.886+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f8290004e60 con 0x7f82980b0770 2024-03-29T10:53:09.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.887+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f8290084c90 con 0x7f82980b0770 2024-03-29T10:53:09.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.887+0000 7f828dffb640 10 client.4281.objecter ms_dispatch 0x7f8298000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:09.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.887+0000 7f828dffb640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:09.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.887+0000 7f828dffb640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:09.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.887+0000 7f828dffb640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T10:53:09.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.887+0000 7f829d733640 10 monclient: get_auth_request con 0x7f827c06ded0 auth_method 0 2024-03-29T10:53:09.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.887+0000 7f828dffb640 10 client.4281.objecter ms_handle_connect 0x7f827c06ded0 2024-03-29T10:53:09.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.903+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8290050570 con 0x7f82980b0770 2024-03-29T10:53:09.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.903+0000 7f828dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:09.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:09.903+0000 7f828dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:10.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.307+0000 7f829f997640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:10.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.307+0000 7f829f997640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:10.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.307+0000 7f829f997640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:10.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.307+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8278005330 con 0x7f82980b0770 2024-03-29T10:53:10.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.308+0000 7f828dffb640 1 -- 172.21.15.155:0/2258108620 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f82900524a0 con 0x7f82980b0770 2024-03-29T10:53:10.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.308+0000 7f828dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:10.290 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.308+0000 7f828dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:10.291 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:10.291 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:10.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 10 librados: watch_flush enter 2024-03-29T10:53:10.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 10 librados: watch_flush exit 2024-03-29T10:53:10.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T10:53:10.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 10 client.4281.objecter successfully canceled tick 2024-03-29T10:53:10.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 >> v1:172.21.15.183:6816/1412921548 conn(0x7f827c06ded0 legacy=0x7f827c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 10 monclient: shutdown 2024-03-29T10:53:10.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:10.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.313+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 >> v1:172.21.15.155:6789/0 conn(0x7f82980b0770 legacy=0x7f8298126c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:10.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.314+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 shutdown_connections 2024-03-29T10:53:10.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.314+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 >> 172.21.15.155:0/2258108620 conn(0x7f829806b660 msgr2=0x7f8298149480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:10.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.314+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 shutdown_connections 2024-03-29T10:53:10.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.314+0000 7f829f997640 1 -- 172.21.15.155:0/2258108620 wait complete. 2024-03-29T10:53:10.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:10.314+0000 7f829f997640 1 librados: shutdown 2024-03-29T10:53:10.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:17.119 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.271+0000 7fd7ea1f4640 10 monclient: get_monmap_and_config 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.271+0000 7fd7ea1f4640 10 monclient: build_initial_monmap 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.271+0000 7fd7ea1f4640 10 monclient: monmap: 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:17.272570+0000 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:17.272570+0000 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:17.253 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:17.254 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:17.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.272+0000 7fd7ea1f4640 1 Processor -- start 2024-03-29T10:53:17.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.273+0000 7fd7ea1f4640 1 -- start start 2024-03-29T10:53:17.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.273+0000 7fd7ea1f4640 10 monclient: init 2024-03-29T10:53:17.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:17.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:17.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd7e41457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:17.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd7e413f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:17.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd7e40a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:17.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e40aaef0 con 0x7fd7e413f650 2024-03-29T10:53:17.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e4145bd0 con 0x7fd7e41457f0 2024-03-29T10:53:17.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e4145eb0 con 0x7fd7e40a9d70 2024-03-29T10:53:17.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:17.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7ea1f4640 10 monclient(hunting): authenticate will time out at 900.566266s 2024-03-29T10:53:17.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7e3fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd7e40a9d70 0x7fd7e414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59336/0 (socket says 172.21.15.155:59336) 2024-03-29T10:53:17.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.274+0000 7fd7e3fff640 1 -- 172.21.15.155:0/3290059503 learned_addr learned my addr 172.21.15.155:0/3290059503 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484699773 0 0) 0x7fd7e4145bd0 con 0x7fd7e41457f0 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7cc003520 con 0x7fd7e41457f0 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429690404 0 0) 0x7fd7e4145eb0 con 0x7fd7e40a9d70 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e4145bd0 con 0x7fd7e40a9d70 2024-03-29T10:53:17.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196316406 0 0) 0x7fd7cc003520 con 0x7fd7e41457f0 2024-03-29T10:53:17.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 >> v1:172.21.15.155:6790/0 conn(0x7fd7e40a9d70 legacy=0x7fd7e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:17.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.275+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 >> v1:172.21.15.155:6789/0 conn(0x7fd7e413f650 legacy=0x7fd7e40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:17.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 1 monclient: found mon.noname-c 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e4146340 con 0x7fd7e41457f0 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd7d4001690 con 0x7fd7e41457f0 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: dump: 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:17.259 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7ea1f4640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7ea1f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7d4001870 con 0x7fd7e41457f0 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 1 -- 172.21.15.155:0/3290059503 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd7d4001c80 con 0x7fd7e41457f0 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7ea1f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:17.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7e27fc640 10 monclient: dump: 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:17.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.277+0000 7fd7ea1f4640 4 set_mon_vals no callback set 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.278+0000 7fd7ea1f4640 10 monclient: shutdown 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.278+0000 7fd7ea1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.278+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3290059503 >> v1:172.21.15.183:6789/0 conn(0x7fd7e41457f0 legacy=0x7fd7e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.278+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3290059503 shutdown_connections 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.278+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3290059503 >> 172.21.15.155:0/3290059503 conn(0x7fd7e4066a00 msgr2=0x7fd7e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.279+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3290059503 shutdown_connections 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.279+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3290059503 wait complete. 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.279+0000 7fd7ea1f4640 10 monclient: build_initial_monmap 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.279+0000 7fd7ea1f4640 10 monclient: monmap: 2024-03-29T10:53:17.265 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.280+0000 7fd7ea1f4640 1 librados: starting msgr at 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.280+0000 7fd7ea1f4640 1 librados: starting objecter 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.280+0000 7fd7ea1f4640 1 Processor -- start 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.280+0000 7fd7ea1f4640 1 -- start start 2024-03-29T10:53:17.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.280+0000 7fd7ea1f4640 1 librados: setting wanted keys 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.280+0000 7fd7ea1f4640 1 librados: calling monclient init 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.280+0000 7fd7ea1f4640 10 monclient: init 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd7e40a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd7e413f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd7e41457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e41dc6f0 con 0x7fd7e40a9d70 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e4071b00 con 0x7fd7e41457f0 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e41dc1f0 con 0x7fd7e413f650 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.281+0000 7fd7ea1f4640 10 monclient(hunting): authenticate will time out at 900.573375s 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7e2ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd7e413f650 0x7fd7e406de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33132/0 (socket says 172.21.15.155:33132) 2024-03-29T10:53:17.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7e2ffd640 1 -- 172.21.15.155:0/3303538354 learned_addr learned my addr 172.21.15.155:0/3303538354 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 10 client.?.objecter ms_handle_connect 0x7fd7e413f650 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520491543 0 0) 0x7fd7e41dc1f0 con 0x7fd7e413f650 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.282+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d0004eb0 con 0x7fd7e413f650 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180119826 0 0) 0x7fd7d0004eb0 con 0x7fd7e413f650 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7d0005300 con 0x7fd7e413f650 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd7d8001920 con 0x7fd7e413f650 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:17.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 10 monclient(hunting): dump: 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 786255473 0 0) 0x7fd7d0005300 con 0x7fd7e413f650 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 >> v1:172.21.15.183:6789/0 conn(0x7fd7e41457f0 legacy=0x7fd7e41e6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:17.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.283+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 >> v1:172.21.15.155:6789/0 conn(0x7fd7e40a9d70 legacy=0x7fd7e41db7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:17.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:17.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 1 monclient: found mon.c 2024-03-29T10:53:17.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e41dbed0 con 0x7fd7e413f650 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: _finish_auth 0 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 10 monclient: _renew_subs 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e4142010 con 0x7fd7e413f650 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7d8001ba0 con 0x7fd7e413f650 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 10 monclient: _renew_subs 2024-03-29T10:53:17.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd7d8003a40 con 0x7fd7e413f650 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e406e5a0 con 0x7fd7e413f650 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7cbfff640 10 monclient: dump: 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:17.274 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7e1ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.285+0000 7fd7ea1f4640 1 librados: init done 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.286+0000 7fd7ea1f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.286+0000 7fd7ea1f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.286+0000 7fd7ea1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.286+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e41dc380 con 0x7fd7e413f650 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.287+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fd7d8003ba0 con 0x7fd7e413f650 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.287+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fd7d8084ac0 con 0x7fd7e413f650 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.287+0000 7fd7cbfff640 10 client.4286.objecter ms_dispatch 0x7fd7e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.287+0000 7fd7cbfff640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.287+0000 7fd7cbfff640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.288+0000 7fd7cbfff640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T10:53:17.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.288+0000 7fd7e37fe640 10 monclient: get_auth_request con 0x7fd7d006de90 auth_method 0 2024-03-29T10:53:17.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.288+0000 7fd7cbfff640 10 client.4286.objecter ms_handle_connect 0x7fd7d006de90 2024-03-29T10:53:17.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.294+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7d8050400 con 0x7fd7e413f650 2024-03-29T10:53:17.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.294+0000 7fd7cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:17.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.294+0000 7fd7cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:17.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.596+0000 7fd7ea1f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:17.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.596+0000 7fd7ea1f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:17.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.596+0000 7fd7ea1f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:17.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.596+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7e41dc060 con 0x7fd7e413f650 2024-03-29T10:53:17.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.597+0000 7fd7cbfff640 1 -- 172.21.15.155:0/3303538354 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fd7d8084170 con 0x7fd7e413f650 2024-03-29T10:53:17.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.597+0000 7fd7cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:17.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.597+0000 7fd7cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:17.583 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:17.583 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:17.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.599+0000 7fd7ea1f4640 10 librados: watch_flush enter 2024-03-29T10:53:17.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.599+0000 7fd7ea1f4640 10 librados: watch_flush exit 2024-03-29T10:53:17.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.599+0000 7fd7ea1f4640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T10:53:17.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.599+0000 7fd7ea1f4640 10 client.4286.objecter successfully canceled tick 2024-03-29T10:53:17.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.600+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 >> v1:172.21.15.183:6816/1412921548 conn(0x7fd7d006de90 legacy=0x7fd7d0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:17.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.600+0000 7fd7ea1f4640 10 monclient: shutdown 2024-03-29T10:53:17.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.600+0000 7fd7ea1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:17.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.600+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 >> v1:172.21.15.155:6790/0 conn(0x7fd7e413f650 legacy=0x7fd7e406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:17.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.600+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 shutdown_connections 2024-03-29T10:53:17.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.600+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 >> 172.21.15.155:0/3303538354 conn(0x7fd7e4066a00 msgr2=0x7fd7e40b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:17.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.601+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 shutdown_connections 2024-03-29T10:53:17.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.601+0000 7fd7ea1f4640 1 -- 172.21.15.155:0/3303538354 wait complete. 2024-03-29T10:53:17.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:17.601+0000 7fd7ea1f4640 1 librados: shutdown 2024-03-29T10:53:17.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:24.404 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.568+0000 7f861366e640 10 monclient: get_monmap_and_config 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.568+0000 7f861366e640 10 monclient: build_initial_monmap 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.568+0000 7f861366e640 10 monclient: monmap: 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:24.569323+0000 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:24.569323+0000 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:24.551 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:24.552 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:24.552 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:24.552 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:24.552 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.570+0000 7f861366e640 1 Processor -- start 2024-03-29T10:53:24.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.570+0000 7f861366e640 1 -- start start 2024-03-29T10:53:24.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.570+0000 7f861366e640 10 monclient: init 2024-03-29T10:53:24.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:24.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:24.554 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 10 monclient(hunting): picked mon.noname-a con 0x7f860c0b4100 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 10 monclient(hunting): picked mon.noname-b con 0x7f860c13ee90 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 10 monclient(hunting): picked mon.noname-c con 0x7f860c0af900 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c0afd10 con 0x7f860c0b4100 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c0af6f0 con 0x7f860c0af900 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c0aaf60 con 0x7f860c13ee90 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.571+0000 7f861366e640 10 monclient(hunting): authenticate will time out at 907.863460s 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8611c0b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f860c0af900 0x7f860c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:49812/0 (socket says 172.21.15.155:49812) 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8611c0b640 1 -- 172.21.15.155:0/2694504812 learned_addr learned my addr 172.21.15.155:0/2694504812 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103530305 0 0) 0x7f860c0af6f0 con 0x7f860c0af900 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8603fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8603fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8603fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:24.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8603fff640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:53:24.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.572+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8003520 con 0x7f860c0af900 2024-03-29T10:53:24.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.573+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 750219353 0 0) 0x7f85f8003520 con 0x7f860c0af900 2024-03-29T10:53:24.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.573+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 >> v1:172.21.15.155:6790/0 conn(0x7f860c13ee90 legacy=0x7f860c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:24.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.575+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 >> v1:172.21.15.155:6789/0 conn(0x7f860c0b4100 legacy=0x7f860c0a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 1 monclient: found mon.noname-c 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c13e970 con 0x7f860c0af900 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f861366e640 5 monclient: authenticate success, global_id 4276 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f861366e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8608001690 con 0x7f860c0af900 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:24.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f8603fff640 10 monclient: dump: 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:24.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.578+0000 7f861366e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:24.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8608001860 con 0x7f860c0af900 2024-03-29T10:53:24.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f8603fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:24.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f861366e640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:24.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f861366e640 4 set_mon_vals no callback set 2024-03-29T10:53:24.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f8603fff640 1 -- 172.21.15.155:0/2694504812 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8608001cb0 con 0x7f860c0af900 2024-03-29T10:53:24.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f8603fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f8603fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.579+0000 7f8603fff640 10 monclient: dump: 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:24.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.580+0000 7f861366e640 10 monclient: shutdown 2024-03-29T10:53:24.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.580+0000 7f861366e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:24.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.580+0000 7f861366e640 1 -- 172.21.15.155:0/2694504812 >> v1:172.21.15.183:6789/0 conn(0x7f860c0af900 legacy=0x7f860c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:24.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.580+0000 7f861366e640 1 -- 172.21.15.155:0/2694504812 shutdown_connections 2024-03-29T10:53:24.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.580+0000 7f861366e640 1 -- 172.21.15.155:0/2694504812 >> 172.21.15.155:0/2694504812 conn(0x7f860c0b35e0 msgr2=0x7f860c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:24.564 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.583+0000 7f861366e640 1 -- 172.21.15.155:0/2694504812 shutdown_connections 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.584+0000 7f861366e640 1 -- 172.21.15.155:0/2694504812 wait complete. 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.585+0000 7f861366e640 10 monclient: build_initial_monmap 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.585+0000 7f861366e640 10 monclient: monmap: 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:24.569 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.586+0000 7f861366e640 1 librados: starting msgr at 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.586+0000 7f861366e640 1 librados: starting objecter 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.586+0000 7f861366e640 1 Processor -- start 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.586+0000 7f861366e640 1 -- start start 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.586+0000 7f861366e640 1 librados: setting wanted keys 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.586+0000 7f861366e640 1 librados: calling monclient init 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.586+0000 7f861366e640 10 monclient: init 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 10 monclient(hunting): picked mon.noname-a con 0x7f860c0af900 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 10 monclient(hunting): picked mon.noname-b con 0x7f860c0b4100 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:24.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 10 monclient(hunting): picked mon.noname-c con 0x7f860c13ee90 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c14c750 con 0x7f860c0af900 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c12ccc0 con 0x7f860c0b4100 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c12cf30 con 0x7f860c13ee90 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.587+0000 7f861366e640 10 monclient(hunting): authenticate will time out at 907.879438s 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.588+0000 7f8610c09640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f860c0b4100 0x7f860c12b6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:50350/0 (socket says 172.21.15.155:50350) 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.588+0000 7f8610c09640 1 -- 172.21.15.155:0/3058650371 learned_addr learned my addr 172.21.15.155:0/3058650371 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:24.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.588+0000 7f8601ffb640 10 client.?.objecter ms_handle_connect 0x7f860c0b4100 2024-03-29T10:53:24.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.588+0000 7f8601ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495225614 0 0) 0x7f860c12ccc0 con 0x7f860c0b4100 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0004eb0 con 0x7f860c0b4100 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715079315 0 0) 0x7f85f0004eb0 con 0x7f860c0b4100 2024-03-29T10:53:24.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f860c12ccc0 con 0x7f860c0b4100 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8604001b50 con 0x7f860c0b4100 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:24.578 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.589+0000 7f8601ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.590+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 844743679 0 0) 0x7f860c12ccc0 con 0x7f860c0b4100 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.590+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 >> v1:172.21.15.155:6790/0 conn(0x7f860c13ee90 legacy=0x7f860c1e1fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 >> v1:172.21.15.155:6789/0 conn(0x7f860c0af900 legacy=0x7f860c071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 1 monclient: found mon.b 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:24.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c12d210 con 0x7f860c0b4100 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 5 monclient: authenticate success, global_id 4279 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 10 monclient: _renew_subs 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c12bfd0 con 0x7f860c0b4100 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 10 monclient: _renew_subs 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c12ab40 con 0x7f860c0b4100 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f861366e640 1 librados: init done 2024-03-29T10:53:24.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8604001cb0 con 0x7f860c0b4100 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f8604004dc0 con 0x7f860c0b4100 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.591+0000 7f8601ffb640 10 monclient: dump: 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:24.581 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.592+0000 7f8602ffd640 4 set_mon_vals no callback set 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f861366e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f861366e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f861366e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec005080 con 0x7f860c0b4100 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f860401ad90 con 0x7f860c0b4100 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f8604084df0 con 0x7f860c0b4100 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f8601ffb640 10 client.4279.objecter ms_dispatch 0x7f860c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f8601ffb640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:24.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f8601ffb640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:24.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.594+0000 7f8601ffb640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T10:53:24.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.595+0000 7f861140a640 10 monclient: get_auth_request con 0x7f85f006dcf0 auth_method 0 2024-03-29T10:53:24.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.595+0000 7f8601ffb640 10 client.4279.objecter ms_handle_connect 0x7f85f006dcf0 2024-03-29T10:53:24.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.604+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8604050500 con 0x7f860c0b4100 2024-03-29T10:53:24.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.604+0000 7f8601ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:24.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.604+0000 7f8601ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:24.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.925+0000 7f861366e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:24.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.925+0000 7f861366e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:24.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.925+0000 7f861366e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:24.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.925+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85ec005330 con 0x7f860c0b4100 2024-03-29T10:53:24.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.927+0000 7f8601ffb640 1 -- 172.21.15.155:0/3058650371 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f8604043b80 con 0x7f860c0b4100 2024-03-29T10:53:24.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.927+0000 7f8601ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:24.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.927+0000 7f8601ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:24.909 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:24.909 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 10 librados: watch_flush enter 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 10 librados: watch_flush exit 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 10 client.4279.objecter successfully canceled tick 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 >> v1:172.21.15.183:6816/1412921548 conn(0x7f85f006dcf0 legacy=0x7f85f00701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 10 monclient: shutdown 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 >> v1:172.21.15.183:6789/0 conn(0x7f860c0b4100 legacy=0x7f860c12b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:24.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 shutdown_connections 2024-03-29T10:53:24.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.931+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 >> 172.21.15.155:0/3058650371 conn(0x7f860c0b35e0 msgr2=0x7f860c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:24.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.932+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 shutdown_connections 2024-03-29T10:53:24.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.932+0000 7f861366e640 1 -- 172.21.15.155:0/3058650371 wait complete. 2024-03-29T10:53:24.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:24.932+0000 7f861366e640 1 librados: shutdown 2024-03-29T10:53:24.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:31.738 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:31.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.915+0000 7f11f7fff640 10 monclient: get_monmap_and_config 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.915+0000 7f11f7fff640 10 monclient: build_initial_monmap 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.915+0000 7f11f7fff640 10 monclient: monmap: 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:31.915952+0000 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:31.915952+0000 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:31.898 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:31.899 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 1 Processor -- start 2024-03-29T10:53:31.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 1 -- start start 2024-03-29T10:53:31.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient: init 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f11f80a9750 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f11f80ae220 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f11f813b9d0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f8142490 con 0x7f11f80a9750 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80ac130 con 0x7f11f80ae220 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f80ac410 con 0x7f11f813b9d0 2024-03-29T10:53:31.903 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.917+0000 7f11f7fff640 10 monclient(hunting): authenticate will time out at 915.209867s 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.921+0000 7f11f77fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f11f813b9d0 0x7f11f8149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34317/0 (socket says 172.21.15.155:34317) 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.921+0000 7f11f77fe640 1 -- 172.21.15.155:0/4286521746 learned_addr learned my addr 172.21.15.155:0/4286521746 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.922+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201491892 0 0) 0x7f11f80ac410 con 0x7f11f813b9d0 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.922+0000 7f11f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.922+0000 7f11f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.922+0000 7f11f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.922+0000 7f11f5ffb640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:53:31.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.922+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e0003520 con 0x7f11f813b9d0 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571844186 0 0) 0x7f11e0003520 con 0x7f11f813b9d0 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 >> v1:172.21.15.183:6789/0 conn(0x7f11f80ae220 legacy=0x7f11f813b2c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 >> v1:172.21.15.155:6789/0 conn(0x7f11f80a9750 legacy=0x7f11f8141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f80afa40 con 0x7f11f813b9d0 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:31.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f11e8001a10 con 0x7f11f813b9d0 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f5ffb640 10 monclient: dump: 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:31.907 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f7fff640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:53:31.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.925+0000 7f11f7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:31.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.927+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11e8001b70 con 0x7f11f813b9d0 2024-03-29T10:53:31.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.927+0000 7f11f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.927+0000 7f11f5ffb640 1 -- 172.21.15.155:0/4286521746 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f11e8001fc0 con 0x7f11f813b9d0 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.927+0000 7f11f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.927+0000 7f11f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.927+0000 7f11f5ffb640 10 monclient: dump: 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:31.909 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.928+0000 7f11f7fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:31.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.928+0000 7f11f7fff640 4 set_mon_vals no callback set 2024-03-29T10:53:31.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.929+0000 7f11f7fff640 10 monclient: shutdown 2024-03-29T10:53:31.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.929+0000 7f11f7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:31.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.929+0000 7f11f7fff640 1 -- 172.21.15.155:0/4286521746 >> v1:172.21.15.155:6790/0 conn(0x7f11f813b9d0 legacy=0x7f11f8149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:31.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.930+0000 7f11f7fff640 1 -- 172.21.15.155:0/4286521746 shutdown_connections 2024-03-29T10:53:31.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.930+0000 7f11f7fff640 1 -- 172.21.15.155:0/4286521746 >> 172.21.15.155:0/4286521746 conn(0x7f11f80673e0 msgr2=0x7f11f80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:31.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.932+0000 7f11f7fff640 1 -- 172.21.15.155:0/4286521746 shutdown_connections 2024-03-29T10:53:31.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.932+0000 7f11f7fff640 1 -- 172.21.15.155:0/4286521746 wait complete. 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.933+0000 7f11f7fff640 10 monclient: build_initial_monmap 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.933+0000 7f11f7fff640 10 monclient: monmap: 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.933+0000 7f11f7fff640 1 librados: starting msgr at 2024-03-29T10:53:31.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.933+0000 7f11f7fff640 1 librados: starting objecter 2024-03-29T10:53:31.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.933+0000 7f11f7fff640 1 Processor -- start 2024-03-29T10:53:31.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.934+0000 7f11f7fff640 1 -- start start 2024-03-29T10:53:31.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.934+0000 7f11f7fff640 1 librados: setting wanted keys 2024-03-29T10:53:31.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.934+0000 7f11f7fff640 1 librados: calling monclient init 2024-03-29T10:53:31.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.934+0000 7f11f7fff640 10 monclient: init 2024-03-29T10:53:31.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.935+0000 7f11f7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:31.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.935+0000 7f11f7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f11f80a9750 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f11f80ae220 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f11f813b9d0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f81e45c0 con 0x7f11f80a9750 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f813d020 con 0x7f11f80ae220 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f813d300 con 0x7f11f813b9d0 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:31.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.937+0000 7f11f7fff640 10 monclient(hunting): authenticate will time out at 915.229362s 2024-03-29T10:53:31.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.942+0000 7f11f67fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f11f80ae220 0x7f11f80b1410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:44467/0 (socket says 172.21.15.155:44467) 2024-03-29T10:53:31.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.942+0000 7f11f77fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f11f813b9d0 0x7f11f81dacd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34451/0 (socket says 172.21.15.155:34451) 2024-03-29T10:53:31.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.942+0000 7f11f77fe640 1 -- 172.21.15.155:0/1440291177 learned_addr learned my addr 172.21.15.155:0/1440291177 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:31.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.942+0000 7f11dffff640 10 client.?.objecter ms_handle_connect 0x7f11f813b9d0 2024-03-29T10:53:31.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.942+0000 7f11dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:31.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.942+0000 7f11dffff640 10 client.?.objecter ms_handle_connect 0x7f11f80a9750 2024-03-29T10:53:31.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.942+0000 7f11dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 client.?.objecter ms_handle_connect 0x7f11f80ae220 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790529265 0 0) 0x7f11f81e45c0 con 0x7f11f80a9750 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:53:31.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e4004eb0 con 0x7f11f80a9750 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846244247 0 0) 0x7f11f813d020 con 0x7f11f80ae220 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f81e45c0 con 0x7f11f80ae220 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187402725 0 0) 0x7f11f813d300 con 0x7f11f813b9d0 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:53:31.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.943+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f813d020 con 0x7f11f813b9d0 2024-03-29T10:53:31.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4193595753 0 0) 0x7f11f81e45c0 con 0x7f11f80ae220 2024-03-29T10:53:31.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f813d300 con 0x7f11f80ae220 2024-03-29T10:53:31.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f11ec001770 con 0x7f11f80ae220 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 10 monclient(hunting): dump: 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:31.933 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 616086402 0 0) 0x7f11e4004eb0 con 0x7f11f80a9750 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f81e45c0 con 0x7f11f80a9750 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f11f0001510 con 0x7f11f80a9750 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.947+0000 7f11dffff640 10 monclient(hunting): dump: 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:31.934 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2937048930 0 0) 0x7f11f813d020 con 0x7f11f813b9d0 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11e4004eb0 con 0x7f11f813b9d0 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f11e8001e80 con 0x7f11f813b9d0 2024-03-29T10:53:31.935 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 10 monclient(hunting): dump: 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.936 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3945072311 0 0) 0x7f11f81e45c0 con 0x7f11f80a9750 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 >> v1:172.21.15.155:6790/0 conn(0x7f11f813b9d0 legacy=0x7f11f81dacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 >> v1:172.21.15.183:6789/0 conn(0x7f11f80ae220 legacy=0x7f11f80b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 monclient: found mon.a 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f80b1b20 con 0x7f11f80a9750 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 10 monclient: _finish_auth 0 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.948+0000 7f11dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 5 monclient: authenticate success, global_id 4284 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 10 monclient: _renew_subs 2024-03-29T10:53:31.937 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f81db3e0 con 0x7f11f80a9750 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 10 monclient: _renew_subs 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f81e4c30 con 0x7f11f80a9750 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.949+0000 7f11f7fff640 1 librados: init done 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.950+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11f0001a20 con 0x7f11f80a9750 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.950+0000 7f11dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.950+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f11f0003910 con 0x7f11f80a9750 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.950+0000 7f11dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:31.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.950+0000 7f11dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.950+0000 7f11dffff640 10 monclient: dump: 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:31.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.950+0000 7f11f57fa640 4 set_mon_vals no callback set 2024-03-29T10:53:31.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.952+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f11f00198e0 con 0x7f11f80a9750 2024-03-29T10:53:31.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.952+0000 7f11f7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:31.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.952+0000 7f11f7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:31.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.952+0000 7f11f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:31.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.952+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d8005080 con 0x7f11f80a9750 2024-03-29T10:53:31.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.953+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f11f0053600 con 0x7f11f80a9750 2024-03-29T10:53:31.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.953+0000 7f11dffff640 10 client.4284.objecter ms_dispatch 0x7f11f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:31.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.953+0000 7f11dffff640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:31.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.953+0000 7f11dffff640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:31.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.953+0000 7f11dffff640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T10:53:31.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.956+0000 7f11f67fc640 10 monclient: get_auth_request con 0x7f11e406e300 auth_method 0 2024-03-29T10:53:31.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.957+0000 7f11dffff640 10 client.4284.objecter ms_handle_connect 0x7f11e406e300 2024-03-29T10:53:31.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.963+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11f004f8a0 con 0x7f11f80a9750 2024-03-29T10:53:31.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.963+0000 7f11dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:31.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:31.963+0000 7f11dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:32.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.298+0000 7f11f7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:32.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.298+0000 7f11f7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:32.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.298+0000 7f11f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:32.279 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.298+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11d8005330 con 0x7f11f80a9750 2024-03-29T10:53:32.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.299+0000 7f11dffff640 1 -- 172.21.15.155:0/1440291177 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f11f004f8a0 con 0x7f11f80a9750 2024-03-29T10:53:32.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.299+0000 7f11dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:32.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.299+0000 7f11dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:32.283 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:32.283 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:32.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.303+0000 7f11f7fff640 10 librados: watch_flush enter 2024-03-29T10:53:32.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 10 librados: watch_flush exit 2024-03-29T10:53:32.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T10:53:32.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 10 client.4284.objecter successfully canceled tick 2024-03-29T10:53:32.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 >> v1:172.21.15.183:6816/1412921548 conn(0x7f11e406e300 legacy=0x7f11e40707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 10 monclient: shutdown 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 >> v1:172.21.15.155:6789/0 conn(0x7f11f80a9750 legacy=0x7f11f81e3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 shutdown_connections 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.305+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 >> 172.21.15.155:0/1440291177 conn(0x7f11f80673e0 msgr2=0x7f11f80b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.306+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 shutdown_connections 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.306+0000 7f11f7fff640 1 -- 172.21.15.155:0/1440291177 wait complete. 2024-03-29T10:53:32.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:32.307+0000 7f11f7fff640 1 librados: shutdown 2024-03-29T10:53:32.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:39.118 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.290+0000 7f7965094640 10 monclient: get_monmap_and_config 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.290+0000 7f7965094640 10 monclient: build_initial_monmap 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.290+0000 7f7965094640 10 monclient: monmap: 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:39.290768+0000 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:39.290768+0000 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:39.271 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:39.272 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:39.272 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:39.272 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.292+0000 7f7965094640 1 Processor -- start 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.292+0000 7f7965094640 1 -- start start 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.292+0000 7f7965094640 10 monclient: init 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 10 monclient(hunting): picked mon.noname-c con 0x7f7958006e30 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 10 monclient(hunting): picked mon.noname-a con 0x7f7958035e00 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:39.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 10 monclient(hunting): picked mon.noname-b con 0x7f79580c9670 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:39.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958035970 con 0x7f7958035e00 2024-03-29T10:53:39.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958036280 con 0x7f7958006e30 2024-03-29T10:53:39.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958035640 con 0x7f79580c9670 2024-03-29T10:53:39.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:39.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.293+0000 7f7965094640 10 monclient(hunting): authenticate will time out at 922.585222s 2024-03-29T10:53:39.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.296+0000 7f795f7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7958035e00 0x7f79580c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:40269/0 (socket says 172.21.15.155:40269) 2024-03-29T10:53:39.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.296+0000 7f795f7fe640 1 -- 172.21.15.155:0/212516505 learned_addr learned my addr 172.21.15.155:0/212516505 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:39.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.300+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916589768 0 0) 0x7f7958035970 con 0x7f7958035e00 2024-03-29T10:53:39.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.300+0000 7f795effd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:39.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.300+0000 7f795effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:39.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.300+0000 7f795effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:39.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.300+0000 7f795effd640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:53:39.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.300+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7950003520 con 0x7f7958035e00 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468233272 0 0) 0x7f7950003520 con 0x7f7958035e00 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 >> v1:172.21.15.155:6790/0 conn(0x7f79580c9670 legacy=0x7f79580d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 >> v1:172.21.15.183:6789/0 conn(0x7f7958006e30 legacy=0x7f79580cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 1 monclient: found mon.noname-a 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958034e20 con 0x7f7958035e00 2024-03-29T10:53:39.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: _finish_auth 0 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f7965094640 5 monclient: authenticate success, global_id 4287 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f7965094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f796004f730 con 0x7f7958035e00 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: dump: 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:39.284 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f7965094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 1 -- 172.21.15.155:0/212516505 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79600008d0 con 0x7f7958035e00 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f795effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f7965094640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.301+0000 7f7965094640 4 set_mon_vals no callback set 2024-03-29T10:53:39.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.302+0000 7f7965094640 10 monclient: shutdown 2024-03-29T10:53:39.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.302+0000 7f7965094640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:39.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.302+0000 7f7965094640 1 -- 172.21.15.155:0/212516505 >> v1:172.21.15.155:6789/0 conn(0x7f7958035e00 legacy=0x7f79580c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:39.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.302+0000 7f7965094640 1 -- 172.21.15.155:0/212516505 shutdown_connections 2024-03-29T10:53:39.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.302+0000 7f7965094640 1 -- 172.21.15.155:0/212516505 >> 172.21.15.155:0/212516505 conn(0x7f7958005c00 msgr2=0x7f7958006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:39.289 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.308+0000 7f7965094640 1 -- 172.21.15.155:0/212516505 shutdown_connections 2024-03-29T10:53:39.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.310+0000 7f7965094640 1 -- 172.21.15.155:0/212516505 wait complete. 2024-03-29T10:53:39.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.311+0000 7f7965094640 10 monclient: build_initial_monmap 2024-03-29T10:53:39.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.311+0000 7f7965094640 10 monclient: monmap: 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:39.293 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:39.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.311+0000 7f7965094640 1 librados: starting msgr at 2024-03-29T10:53:39.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.311+0000 7f7965094640 1 librados: starting objecter 2024-03-29T10:53:39.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.311+0000 7f7965094640 1 Processor -- start 2024-03-29T10:53:39.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.312+0000 7f7965094640 1 -- start start 2024-03-29T10:53:39.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.312+0000 7f7965094640 1 librados: setting wanted keys 2024-03-29T10:53:39.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.312+0000 7f7965094640 1 librados: calling monclient init 2024-03-29T10:53:39.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.312+0000 7f7965094640 10 monclient: init 2024-03-29T10:53:39.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:39.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 10 monclient(hunting): picked mon.noname-b con 0x7f7958006e30 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 10 monclient(hunting): picked mon.noname-c con 0x7f7958035e00 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 10 monclient(hunting): picked mon.noname-a con 0x7f79580c9670 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795802f200 con 0x7f79580c9670 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795802f4a0 con 0x7f7958006e30 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958033d20 con 0x7f7958035e00 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:39.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.314+0000 7f7965094640 10 monclient(hunting): authenticate will time out at 922.606253s 2024-03-29T10:53:39.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.318+0000 7f795f7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f7958035e00 0x7f7958033610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33615/0 (socket says 172.21.15.155:33615) 2024-03-29T10:53:39.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.318+0000 7f795f7fe640 1 -- 172.21.15.155:0/4178288229 learned_addr learned my addr 172.21.15.155:0/4178288229 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:39.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.318+0000 7f795ffff640 1 -- 172.21.15.155:0/4178288229 >> v1:172.21.15.183:6789/0 conn(0x7f7958006e30 legacy=0x7f795802f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:53:39.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.318+0000 7f795ffff640 1 -- 172.21.15.155:0/4178288229 >> v1:172.21.15.183:6789/0 conn(0x7f7958006e30 legacy=0x7f795802f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.318+0000 7f795ffff640 1 --1- 172.21.15.155:0/4178288229 >> v1:172.21.15.183:6789/0 conn(0x7f7958006e30 0x7f795802f9c0 :-1 s=OPENED pgs=229328 cs=1 l=1).handle_message read tag failed 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.318+0000 7f795ffff640 1 --1- 172.21.15.155:0/4178288229 >> v1:172.21.15.183:6789/0 conn(0x7f7958006e30 0x7f795802f9c0 :-1 s=OPENED pgs=229328 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 client.?.objecter ms_handle_connect 0x7f7958035e00 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 client.?.objecter ms_handle_connect 0x7f7958006e30 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.183:6789/0 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764812996 0 0) 0x7f7958033d20 con 0x7f7958035e00 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:53:39.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.319+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7944004eb0 con 0x7f7958035e00 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 client.?.objecter ms_handle_connect 0x7f79580c9670 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827025377 0 0) 0x7f795802f200 con 0x7f79580c9670 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7958033d20 con 0x7f79580c9670 2024-03-29T10:53:39.303 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928430023 0 0) 0x7f7944004eb0 con 0x7f7958035e00 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f795802f200 con 0x7f7958035e00 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f796004f400 con 0x7f7958035e00 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): dump: 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:39.304 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:39.305 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:39.305 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.322+0000 7f795cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.323+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2544583173 0 0) 0x7f795802f200 con 0x7f7958035e00 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 >> v1:172.21.15.183:6789/0 conn(0x7f7958006e30 legacy=0x7f795802f9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 >> v1:172.21.15.155:6789/0 conn(0x7f79580c9670 legacy=0x7f795816b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 1 monclient: found mon.c 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958033eb0 con 0x7f7958035e00 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f795cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:39.307 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 10 monclient: _renew_subs 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795802dc40 con 0x7f7958035e00 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 10 monclient: _renew_subs 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f795802d810 con 0x7f7958035e00 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.324+0000 7f7965094640 1 librados: init done 2024-03-29T10:53:39.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.326+0000 7f7965094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:39.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.326+0000 7f7965094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:39.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.326+0000 7f7965094640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:39.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.326+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958007b90 con 0x7f7958035e00 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.328+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f796004fdd0 con 0x7f7958035e00 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.329+0000 7f795cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.329+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7960075710 con 0x7f7958035e00 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.329+0000 7f795cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.329+0000 7f795cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.329+0000 7f795cff9640 10 monclient: dump: 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:39.310 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:39.311 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:39.311 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:39.311 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:39.311 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:39.311 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:39.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.329+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7960067ea0 con 0x7f7958035e00 2024-03-29T10:53:39.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.330+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f79600ff5e0 con 0x7f7958035e00 2024-03-29T10:53:39.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.330+0000 7f795cff9640 10 client.4298.objecter ms_dispatch 0x7f7960000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:39.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.330+0000 7f795cff9640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:39.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.330+0000 7f795cff9640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:39.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.331+0000 7f795cff9640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T10:53:39.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.331+0000 7f795ffff640 10 monclient: get_auth_request con 0x7f794406dca0 auth_method 0 2024-03-29T10:53:39.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.332+0000 7f795e7fc640 4 set_mon_vals no callback set 2024-03-29T10:53:39.316 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.335+0000 7f795cff9640 10 client.4298.objecter ms_handle_connect 0x7f794406dca0 2024-03-29T10:53:39.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.337+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79600caf40 con 0x7f7958035e00 2024-03-29T10:53:39.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.337+0000 7f795cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:39.318 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.338+0000 7f795cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:39.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.664+0000 7f7965094640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:39.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.664+0000 7f7965094640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:39.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.664+0000 7f7965094640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:53:39.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.664+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f795802ddd0 con 0x7f7958035e00 2024-03-29T10:53:39.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.666+0000 7f795cff9640 1 -- 172.21.15.155:0/4178288229 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f79600cce70 con 0x7f7958035e00 2024-03-29T10:53:39.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.666+0000 7f795cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:39.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.666+0000 7f795cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:39.648 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:39.648 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 10 librados: watch_flush enter 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 10 librados: watch_flush exit 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 10 client.4298.objecter successfully canceled tick 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 >> v1:172.21.15.183:6816/1412921548 conn(0x7f794406dca0 legacy=0x7f7944070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 10 monclient: shutdown 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:39.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 >> v1:172.21.15.155:6790/0 conn(0x7f7958035e00 legacy=0x7f7958033610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:39.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 shutdown_connections 2024-03-29T10:53:39.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.670+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 >> 172.21.15.155:0/4178288229 conn(0x7f7958005c00 msgr2=0x7f795803ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:39.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.671+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 shutdown_connections 2024-03-29T10:53:39.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.671+0000 7f7965094640 1 -- 172.21.15.155:0/4178288229 wait complete. 2024-03-29T10:53:39.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:39.671+0000 7f7965094640 1 librados: shutdown 2024-03-29T10:53:39.672 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:46.479 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.650+0000 7f58b9d32640 10 monclient: get_monmap_and_config 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.650+0000 7f58b9d32640 10 monclient: build_initial_monmap 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.650+0000 7f58b9d32640 10 monclient: monmap: 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:46.650867+0000 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:46.650867+0000 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:46.631 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:46.632 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:46.632 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:46.632 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.651+0000 7f58b9d32640 1 Processor -- start 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.651+0000 7f58b9d32640 1 -- start start 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.651+0000 7f58b9d32640 10 monclient: init 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 10 monclient(hunting): picked mon.noname-a con 0x7f58b40b1f60 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 10 monclient(hunting): picked mon.noname-b con 0x7f58b4071eb0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 10 monclient(hunting): picked mon.noname-c con 0x7f58b4143d10 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b406ef80 con 0x7f58b40b1f60 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b406f260 con 0x7f58b4143d10 2024-03-29T10:53:46.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b40b40c0 con 0x7f58b4071eb0 2024-03-29T10:53:46.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:46.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.652+0000 7f58b9d32640 10 monclient(hunting): authenticate will time out at 929.944361s 2024-03-29T10:53:46.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.654+0000 7f58b37fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f58b40b1f60 0x7f58b4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:47023/0 (socket says 172.21.15.155:47023) 2024-03-29T10:53:46.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.654+0000 7f58b37fe640 1 -- 172.21.15.155:0/2173400444 learned_addr learned my addr 172.21.15.155:0/2173400444 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:46.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186269416 0 0) 0x7f58b406ef80 con 0x7f58b40b1f60 2024-03-29T10:53:46.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:46.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:46.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:46.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:53:46.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f589c003520 con 0x7f58b40b1f60 2024-03-29T10:53:46.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3391994534 0 0) 0x7f589c003520 con 0x7f58b40b1f60 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 >> v1:172.21.15.155:6790/0 conn(0x7f58b4071eb0 legacy=0x7f58b4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 >> v1:172.21.15.183:6789/0 conn(0x7f58b4143d10 legacy=0x7f58b4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 monclient: found mon.noname-a 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4155d70 con 0x7f58b40b1f60 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f58ac001a10 con 0x7f58b40b1f60 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:46.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: dump: 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58ac001ca0 con 0x7f58b40b1f60 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:46.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 1 -- 172.21.15.155:0/2173400444 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f58ac002030 con 0x7f58b40b1f60 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.659+0000 7f58b27fc640 10 monclient: dump: 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:46.647 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.660+0000 7f58b9d32640 5 monclient: authenticate success, global_id 4293 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.660+0000 7f58b9d32640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.660+0000 7f58b9d32640 4 set_mon_vals no callback set 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.661+0000 7f58b9d32640 10 monclient: shutdown 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.661+0000 7f58b9d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.661+0000 7f58b9d32640 1 -- 172.21.15.155:0/2173400444 >> v1:172.21.15.155:6789/0 conn(0x7f58b40b1f60 legacy=0x7f58b4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.661+0000 7f58b9d32640 1 -- 172.21.15.155:0/2173400444 shutdown_connections 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.661+0000 7f58b9d32640 1 -- 172.21.15.155:0/2173400444 >> 172.21.15.155:0/2173400444 conn(0x7f58b4066a00 msgr2=0x7f58b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.662+0000 7f58b9d32640 1 -- 172.21.15.155:0/2173400444 shutdown_connections 2024-03-29T10:53:46.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.662+0000 7f58b9d32640 1 -- 172.21.15.155:0/2173400444 wait complete. 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 10 monclient: build_initial_monmap 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 10 monclient: monmap: 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:46.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 1 librados: starting msgr at 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 1 librados: starting objecter 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 1 Processor -- start 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 1 -- start start 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 1 librados: setting wanted keys 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 1 librados: calling monclient init 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.663+0000 7f58b9d32640 10 monclient: init 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 10 monclient(hunting): picked mon.noname-a con 0x7f58b40b1f60 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 10 monclient(hunting): picked mon.noname-b con 0x7f58b4143d10 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 10 monclient(hunting): picked mon.noname-c con 0x7f58b40abca0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b40acd60 con 0x7f58b40b1f60 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b40ab810 con 0x7f58b4143d10 2024-03-29T10:53:46.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b4133750 con 0x7f58b40abca0 2024-03-29T10:53:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:46.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.664+0000 7f58b9d32640 10 monclient(hunting): authenticate will time out at 929.956694s 2024-03-29T10:53:46.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.669+0000 7f58b2ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f58b4143d10 0x7f58b4132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:55293/0 (socket says 172.21.15.155:55293) 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.669+0000 7f58b2ffd640 1 -- 172.21.15.155:0/411017750 learned_addr learned my addr 172.21.15.155:0/411017750 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 10 client.?.objecter ms_handle_connect 0x7f58b4143d10 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081063607 0 0) 0x7f58b40ab810 con 0x7f58b4143d10 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.670+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a0004eb0 con 0x7f58b4143d10 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1288866343 0 0) 0x7f58a0004eb0 con 0x7f58b4143d10 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58b40ab810 con 0x7f58b4143d10 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f58a80019d0 con 0x7f58b4143d10 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:46.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 10 monclient(hunting): dump: 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:46.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 207162805 0 0) 0x7f58b40ab810 con 0x7f58b4143d10 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.672+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 >> v1:172.21.15.155:6790/0 conn(0x7f58b40abca0 legacy=0x7f58b4133040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 >> v1:172.21.15.155:6789/0 conn(0x7f58b40b1f60 legacy=0x7f58b41390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 1 monclient: found mon.b 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b40acf90 con 0x7f58b4143d10 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 monclient: _finish_auth 0 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 5 monclient: authenticate success, global_id 4288 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 10 monclient: _renew_subs 2024-03-29T10:53:46.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b40aa930 con 0x7f58b4143d10 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 10 monclient: _renew_subs 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b41399d0 con 0x7f58b4143d10 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f58b9d32640 1 librados: init done 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 client.4288.objecter ms_handle_connect 0x7f58b40b1f60 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 client.4288.objecter resend_mon_ops 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 client.4288.objecter ms_handle_connect 0x7f58b40abca0 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.675+0000 7f589bfff640 10 client.4288.objecter resend_mon_ops 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.676+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58a8001cd0 con 0x7f58b4143d10 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.676+0000 7f589bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:46.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.676+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f58a8003a60 con 0x7f58b4143d10 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.676+0000 7f589bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.676+0000 7f589bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.676+0000 7f589bfff640 10 monclient: dump: 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:46.661 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.676+0000 7f58b1ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.677+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f58a8003c80 con 0x7f58b4143d10 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f58a8084b50 con 0x7f58b4143d10 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f589bfff640 10 client.4288.objecter ms_dispatch 0x7f58b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f589bfff640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f589bfff640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f58b37fe640 10 monclient: get_auth_request con 0x7f58a006dc30 auth_method 0 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f589bfff640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f589bfff640 10 client.4288.objecter ms_handle_connect 0x7f58a006dc30 2024-03-29T10:53:46.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f58b9d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:46.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f58b9d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:46.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f58b9d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:46.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.678+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5894005080 con 0x7f58b4143d10 2024-03-29T10:53:46.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.688+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58a8050430 con 0x7f58b4143d10 2024-03-29T10:53:46.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.688+0000 7f589bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:46.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:46.688+0000 7f589bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:46.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.012+0000 7f58b9d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:46.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.012+0000 7f58b9d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:46.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.012+0000 7f58b9d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:53:46.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.013+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5894005330 con 0x7f58b4143d10 2024-03-29T10:53:46.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.015+0000 7f589bfff640 1 -- 172.21.15.155:0/411017750 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f58a8052360 con 0x7f58b4143d10 2024-03-29T10:53:46.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.015+0000 7f589bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:46.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.015+0000 7f589bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:47.000 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:47.001 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:47.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.021+0000 7f58b9d32640 10 librados: watch_flush enter 2024-03-29T10:53:47.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.021+0000 7f58b9d32640 10 librados: watch_flush exit 2024-03-29T10:53:47.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.021+0000 7f58b9d32640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T10:53:47.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.021+0000 7f58b9d32640 10 client.4288.objecter successfully canceled tick 2024-03-29T10:53:47.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 >> v1:172.21.15.183:6816/1412921548 conn(0x7f58a006dc30 legacy=0x7f58a00700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 10 monclient: shutdown 2024-03-29T10:53:47.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:47.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 >> v1:172.21.15.183:6789/0 conn(0x7f58b4143d10 legacy=0x7f58b4132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:47.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 shutdown_connections 2024-03-29T10:53:47.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 >> 172.21.15.155:0/411017750 conn(0x7f58b4066a00 msgr2=0x7f58b40b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:47.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 shutdown_connections 2024-03-29T10:53:47.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.024+0000 7f58b9d32640 1 -- 172.21.15.155:0/411017750 wait complete. 2024-03-29T10:53:47.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:47.025+0000 7f58b9d32640 1 librados: shutdown 2024-03-29T10:53:47.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:53.829 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.002+0000 7f2f77dcf640 10 monclient: get_monmap_and_config 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.002+0000 7f2f77dcf640 10 monclient: build_initial_monmap 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.002+0000 7f2f77dcf640 10 monclient: monmap: 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:53:54.004242+0000 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:53:54.004242+0000 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:53:53.988 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.004+0000 7f2f77dcf640 1 Processor -- start 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.005+0000 7f2f77dcf640 1 -- start start 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.005+0000 7f2f77dcf640 10 monclient: init 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f2f700af6c0 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f2f7014f6b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f2f70152980 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700a94e0 con 0x7f2f70152980 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700afb60 con 0x7f2f700af6c0 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700aab80 con 0x7f2f7014f6b0 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:53.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.006+0000 7f2f77dcf640 10 monclient(hunting): authenticate will time out at 937.299060s 2024-03-29T10:53:53.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.010+0000 7f2f75b6b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2f700af6c0 0x7f2f700ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:42143/0 (socket says 172.21.15.155:42143) 2024-03-29T10:53:53.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.010+0000 7f2f75b6b640 1 -- 172.21.15.155:0/3619773136 learned_addr learned my addr 172.21.15.155:0/3619773136 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:53.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873029375 0 0) 0x7f2f700aab80 con 0x7f2f7014f6b0 2024-03-29T10:53:53.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 10 monclient(hunting): _negotiate 2024-03-29T10:53:53.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:53.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:53.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:53:53.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c005730 con 0x7f2f7014f6b0 2024-03-29T10:53:53.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257388614 0 0) 0x7f2f5c005730 con 0x7f2f7014f6b0 2024-03-29T10:53:53.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.011+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 >> v1:172.21.15.183:6789/0 conn(0x7f2f700af6c0 legacy=0x7f2f700ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:53.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.012+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 >> v1:172.21.15.155:6789/0 conn(0x7f2f70152980 legacy=0x7f2f70152e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:53.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: _finish_hunting 0 2024-03-29T10:53:53.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 1 monclient: found mon.noname-b 2024-03-29T10:53:53.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:53.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:53:53.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700aa680 con 0x7f2f7014f6b0 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: _finish_auth 0 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: _check_auth_tickets 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2f6c0015c0 con 0x7f2f7014f6b0 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: dump: 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:53.996 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:53.997 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f77dcf640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f77dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f6c001790 con 0x7f2f7014f6b0 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 1 -- 172.21.15.155:0/3619773136 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2f6c001be0 con 0x7f2f7014f6b0 2024-03-29T10:53:53.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.013+0000 7f2f74b69640 10 monclient: dump: 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:53.999 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.014+0000 7f2f77dcf640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.014+0000 7f2f77dcf640 4 set_mon_vals no callback set 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.014+0000 7f2f77dcf640 10 monclient: shutdown 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.014+0000 7f2f77dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.014+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3619773136 >> v1:172.21.15.155:6790/0 conn(0x7f2f7014f6b0 legacy=0x7f2f7014fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.015+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3619773136 shutdown_connections 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.015+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3619773136 >> 172.21.15.155:0/3619773136 conn(0x7f2f700b35e0 msgr2=0x7f2f700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:54.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.018+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3619773136 shutdown_connections 2024-03-29T10:53:54.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.018+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3619773136 wait complete. 2024-03-29T10:53:54.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.019+0000 7f2f77dcf640 10 monclient: build_initial_monmap 2024-03-29T10:53:54.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.019+0000 7f2f77dcf640 10 monclient: monmap: 2024-03-29T10:53:54.001 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:53:54.001 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:54.001 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:53:54.002 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:53:54.002 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:54.002 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:54.002 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:53:54.002 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:53:54.002 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:53:54.002 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.020+0000 7f2f77dcf640 1 librados: starting msgr at 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.020+0000 7f2f77dcf640 1 librados: starting objecter 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.020+0000 7f2f77dcf640 1 Processor -- start 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.020+0000 7f2f77dcf640 1 -- start start 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.020+0000 7f2f77dcf640 1 librados: setting wanted keys 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.020+0000 7f2f77dcf640 1 librados: calling monclient init 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.020+0000 7f2f77dcf640 10 monclient: init 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.021+0000 7f2f77dcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:54.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.021+0000 7f2f77dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:54.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.021+0000 7f2f77dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f2f700af6c0 addr v1:172.21.15.183:6789/0 2024-03-29T10:53:54.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.022+0000 7f2f77dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f2f70152980 addr v1:172.21.15.155:6790/0 2024-03-29T10:53:54.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.022+0000 7f2f77dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f2f701ecf50 addr v1:172.21.15.155:6789/0 2024-03-29T10:53:54.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.022+0000 7f2f77dcf640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7006fb90 con 0x7f2f701ecf50 2024-03-29T10:53:54.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.022+0000 7f2f77dcf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f701db280 con 0x7f2f700af6c0 2024-03-29T10:53:54.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.022+0000 7f2f77dcf640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f7006e7a0 con 0x7f2f70152980 2024-03-29T10:53:54.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.022+0000 7f2f77dcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:54.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.022+0000 7f2f77dcf640 10 monclient(hunting): authenticate will time out at 937.315495s 2024-03-29T10:53:54.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.026+0000 7f2f75b6b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2f700af6c0 0x7f2f701e7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:42237/0 (socket says 172.21.15.155:42237) 2024-03-29T10:53:54.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.026+0000 7f2f75b6b640 1 -- 172.21.15.155:0/3603014878 learned_addr learned my addr 172.21.15.155:0/3603014878 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:53:54.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 10 client.?.objecter ms_handle_connect 0x7f2f701ecf50 2024-03-29T10:53:54.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:54.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464286625 0 0) 0x7f2f7006fb90 con 0x7f2f701ecf50 2024-03-29T10:53:54.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:54.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:54.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:54.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:53:54.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.027+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f54004eb0 con 0x7f2f701ecf50 2024-03-29T10:53:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 client.?.objecter ms_handle_connect 0x7f2f700af6c0 2024-03-29T10:53:54.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 202898470 0 0) 0x7f2f54004eb0 con 0x7f2f701ecf50 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f7006fb90 con 0x7f2f701ecf50 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 client.?.objecter ms_handle_connect 0x7f2f70152980 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2f68001ad0 con 0x7f2f701ecf50 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): dump: 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:54.011 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175823778 0 0) 0x7f2f701db280 con 0x7f2f700af6c0 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:53:54.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.029+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f54004eb0 con 0x7f2f700af6c0 2024-03-29T10:53:54.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.031+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781885636 0 0) 0x7f2f7006e7a0 con 0x7f2f70152980 2024-03-29T10:53:54.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.031+0000 7f2f667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:54.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.031+0000 7f2f667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:54.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.031+0000 7f2f667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:54.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.031+0000 7f2f667fc640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:53:54.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.031+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f701db280 con 0x7f2f70152980 2024-03-29T10:53:54.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.032+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3244831779 0 0) 0x7f2f701db280 con 0x7f2f70152980 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.032+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f7006e7a0 con 0x7f2f70152980 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.032+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2f6c001aa0 con 0x7f2f70152980 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.032+0000 7f2f667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.032+0000 7f2f667fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.032+0000 7f2f667fc640 10 monclient(hunting): dump: 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:54.015 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:54.016 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:54.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.032+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 189463658 0 0) 0x7f2f54004eb0 con 0x7f2f700af6c0 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f701db280 con 0x7f2f700af6c0 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3270296022 0 0) 0x7f2f7006fb90 con 0x7f2f701ecf50 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 >> v1:172.21.15.155:6790/0 conn(0x7f2f70152980 legacy=0x7f2f701e83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 >> v1:172.21.15.183:6789/0 conn(0x7f2f700af6c0 legacy=0x7f2f701e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 1 monclient: found mon.a 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:54.020 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f701d8e70 con 0x7f2f701ecf50 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 5 monclient: authenticate success, global_id 4299 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 10 monclient: _renew_subs 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f701dbae0 con 0x7f2f701ecf50 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 10 monclient: _renew_subs 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7006ee40 con 0x7f2f701ecf50 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.033+0000 7f2f77dcf640 1 librados: init done 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.035+0000 7f2f77dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.035+0000 7f2f77dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:54.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.035+0000 7f2f77dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.035+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7006e930 con 0x7f2f701ecf50 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.036+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f68002060 con 0x7f2f701ecf50 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.036+0000 7f2f667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.037+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2f68003b10 con 0x7f2f701ecf50 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.037+0000 7f2f667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.037+0000 7f2f667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.037+0000 7f2f667fc640 10 monclient: dump: 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:53:54.022 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:54.023 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:53:54.023 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:53:54.023 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:53:54.023 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:53:54.023 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:53:54.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.038+0000 7f2f67fff640 4 set_mon_vals no callback set 2024-03-29T10:53:54.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.038+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f2f68003cf0 con 0x7f2f701ecf50 2024-03-29T10:53:54.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.039+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f2f68083890 con 0x7f2f701ecf50 2024-03-29T10:53:54.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.039+0000 7f2f667fc640 10 client.4299.objecter ms_dispatch 0x7f2f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:54.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.039+0000 7f2f667fc640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:54.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.039+0000 7f2f667fc640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:54.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.039+0000 7f2f667fc640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T10:53:54.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.043+0000 7f2f75b6b640 10 monclient: get_auth_request con 0x7f2f5406e1f0 auth_method 0 2024-03-29T10:53:54.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.043+0000 7f2f667fc640 10 client.4299.objecter ms_handle_connect 0x7f2f5406e1f0 2024-03-29T10:53:54.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.047+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f6804f1d0 con 0x7f2f701ecf50 2024-03-29T10:53:54.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.047+0000 7f2f667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:54.029 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.047+0000 7f2f667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.367+0000 7f2f77dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.367+0000 7f2f77dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.367+0000 7f2f77dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:53:54.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.367+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f701d9000 con 0x7f2f701ecf50 2024-03-29T10:53:54.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.370+0000 7f2f667fc640 1 -- 172.21.15.155:0/3603014878 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f2f6804f1d0 con 0x7f2f701ecf50 2024-03-29T10:53:54.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.370+0000 7f2f667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:54.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.370+0000 7f2f667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:54.353 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:53:54.353 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:54.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.374+0000 7f2f77dcf640 10 librados: watch_flush enter 2024-03-29T10:53:54.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.378+0000 7f2f77dcf640 10 librados: watch_flush exit 2024-03-29T10:53:54.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.378+0000 7f2f77dcf640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T10:53:54.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.378+0000 7f2f77dcf640 10 client.4299.objecter successfully canceled tick 2024-03-29T10:53:54.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.379+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 >> v1:172.21.15.183:6816/1412921548 conn(0x7f2f5406e1f0 legacy=0x7f2f540706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.379+0000 7f2f77dcf640 10 monclient: shutdown 2024-03-29T10:53:54.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.379+0000 7f2f77dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:54.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.379+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 >> v1:172.21.15.155:6789/0 conn(0x7f2f701ecf50 legacy=0x7f2f701ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:54.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.379+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 shutdown_connections 2024-03-29T10:53:54.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.379+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 >> 172.21.15.155:0/3603014878 conn(0x7f2f700b35e0 msgr2=0x7f2f700aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:54.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.380+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 shutdown_connections 2024-03-29T10:53:54.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.380+0000 7f2f77dcf640 1 -- 172.21.15.155:0/3603014878 wait complete. 2024-03-29T10:53:54.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:53:54.380+0000 7f2f77dcf640 1 librados: shutdown 2024-03-29T10:53:54.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:01.189 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:01.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.358+0000 7f2c492b3640 10 monclient: get_monmap_and_config 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.358+0000 7f2c492b3640 10 monclient: build_initial_monmap 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.358+0000 7f2c492b3640 10 monclient: monmap: 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:01.359736+0000 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:01.359736+0000 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:01.340 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:01.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.358+0000 7f2c492b3640 1 Processor -- start 2024-03-29T10:54:01.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 1 -- start start 2024-03-29T10:54:01.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient: init 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient(hunting): picked mon.noname-b con 0x7f2c441457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient(hunting): picked mon.noname-c con 0x7f2c4413f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient(hunting): picked mon.noname-a con 0x7f2c440a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440aaef0 con 0x7f2c440a9d70 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c44145bd0 con 0x7f2c4413f650 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c44145eb0 con 0x7f2c441457f0 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:01.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.359+0000 7f2c492b3640 10 monclient(hunting): authenticate will time out at 944.652823s 2024-03-29T10:54:01.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.363+0000 7f2c42ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f2c441457f0 0x7f2c4413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40473/0 (socket says 172.21.15.155:40473) 2024-03-29T10:54:01.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.363+0000 7f2c42ffd640 1 -- 172.21.15.155:0/2322646823 learned_addr learned my addr 172.21.15.155:0/2322646823 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:01.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396645693 0 0) 0x7f2c44145eb0 con 0x7f2c441457f0 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c2c003520 con 0x7f2c441457f0 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 362555588 0 0) 0x7f2c2c003520 con 0x7f2c441457f0 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 >> v1:172.21.15.183:6789/0 conn(0x7f2c4413f650 legacy=0x7f2c440b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 >> v1:172.21.15.155:6789/0 conn(0x7f2c440a9d70 legacy=0x7f2c4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 monclient: found mon.noname-b 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:01.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c44146340 con 0x7f2c441457f0 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2c38001690 con 0x7f2c441457f0 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: dump: 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:01.352 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c492b3640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c492b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c38001920 con 0x7f2c441457f0 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 1 -- 172.21.15.155:0/2322646823 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2c38001cb0 con 0x7f2c441457f0 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c41ffb640 10 monclient: dump: 2024-03-29T10:54:01.353 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c492b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.364+0000 7f2c492b3640 4 set_mon_vals no callback set 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.365+0000 7f2c492b3640 10 monclient: shutdown 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.365+0000 7f2c492b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:01.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.365+0000 7f2c492b3640 1 -- 172.21.15.155:0/2322646823 >> v1:172.21.15.155:6790/0 conn(0x7f2c441457f0 legacy=0x7f2c4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:01.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.367+0000 7f2c492b3640 1 -- 172.21.15.155:0/2322646823 shutdown_connections 2024-03-29T10:54:01.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.367+0000 7f2c492b3640 1 -- 172.21.15.155:0/2322646823 >> 172.21.15.155:0/2322646823 conn(0x7f2c44066a00 msgr2=0x7f2c44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:01.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.370+0000 7f2c492b3640 1 -- 172.21.15.155:0/2322646823 shutdown_connections 2024-03-29T10:54:01.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.370+0000 7f2c492b3640 1 -- 172.21.15.155:0/2322646823 wait complete. 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.371+0000 7f2c492b3640 10 monclient: build_initial_monmap 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.371+0000 7f2c492b3640 10 monclient: monmap: 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.372+0000 7f2c492b3640 1 librados: starting msgr at 2024-03-29T10:54:01.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.372+0000 7f2c492b3640 1 librados: starting objecter 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.372+0000 7f2c492b3640 1 Processor -- start 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.372+0000 7f2c492b3640 1 -- start start 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.372+0000 7f2c492b3640 1 librados: setting wanted keys 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.372+0000 7f2c492b3640 1 librados: calling monclient init 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.372+0000 7f2c492b3640 10 monclient: init 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.373+0000 7f2c492b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.373+0000 7f2c492b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.373+0000 7f2c492b3640 10 monclient(hunting): picked mon.noname-a con 0x7f2c440a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:01.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.373+0000 7f2c492b3640 10 monclient(hunting): picked mon.noname-b con 0x7f2c4413f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.374+0000 7f2c492b3640 10 monclient(hunting): picked mon.noname-c con 0x7f2c441457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.374+0000 7f2c492b3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440ac240 con 0x7f2c440a9d70 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.374+0000 7f2c492b3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c440ac4b0 con 0x7f2c4413f650 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.374+0000 7f2c492b3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4413c7e0 con 0x7f2c441457f0 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.374+0000 7f2c492b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.374+0000 7f2c492b3640 10 monclient(hunting): authenticate will time out at 944.667068s 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.376+0000 7f2c42ffd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2c440a9d70 0x7f2c440708c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:55955/0 (socket says 172.21.15.155:55955) 2024-03-29T10:54:01.359 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.377+0000 7f2c42ffd640 1 -- 172.21.15.155:0/1384033716 learned_addr learned my addr 172.21.15.155:0/1384033716 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:01.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.377+0000 7f2c2b7fe640 10 client.?.objecter ms_handle_connect 0x7f2c4413f650 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.378+0000 7f2c2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.378+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847184016 0 0) 0x7f2c440ac4b0 con 0x7f2c4413f650 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.378+0000 7f2c2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.378+0000 7f2c2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.378+0000 7f2c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.378+0000 7f2c2b7fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.378+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c30004eb0 con 0x7f2c4413f650 2024-03-29T10:54:01.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1289284762 0 0) 0x7f2c30004eb0 con 0x7f2c4413f650 2024-03-29T10:54:01.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c440ac4b0 con 0x7f2c4413f650 2024-03-29T10:54:01.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2c3c001bf0 con 0x7f2c4413f650 2024-03-29T10:54:01.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 10 monclient(hunting): dump: 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:01.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 10 client.?.objecter ms_handle_connect 0x7f2c440a9d70 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.379+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 135886002 0 0) 0x7f2c440ac4b0 con 0x7f2c4413f650 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 >> v1:172.21.15.155:6790/0 conn(0x7f2c441457f0 legacy=0x7f2c440b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 >> v1:172.21.15.155:6789/0 conn(0x7f2c440a9d70 legacy=0x7f2c440708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 1 monclient: found mon.b 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:01.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4406f510 con 0x7f2c4413f650 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 5 monclient: authenticate success, global_id 4297 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 10 monclient: _renew_subs 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4413b5d0 con 0x7f2c4413f650 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:01.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 10 monclient: _renew_subs 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4406f0d0 con 0x7f2c4413f650 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c492b3640 1 librados: init done 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c3c001d50 con 0x7f2c4413f650 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.380+0000 7f2c417fa640 4 set_mon_vals no callback set 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.381+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2c3c003c30 con 0x7f2c4413f650 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.381+0000 7f2c2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.382+0000 7f2c2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.382+0000 7f2c2b7fe640 10 monclient: dump: 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:01.366 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.382+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f2c3c003d90 con 0x7f2c4413f650 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.383+0000 7f2c492b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.383+0000 7f2c492b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.383+0000 7f2c492b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:01.367 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.383+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c24005080 con 0x7f2c4413f650 2024-03-29T10:54:01.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.384+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f2c3c083c10 con 0x7f2c4413f650 2024-03-29T10:54:01.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.384+0000 7f2c2b7fe640 10 client.4297.objecter ms_dispatch 0x7f2c44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:01.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.384+0000 7f2c2b7fe640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:01.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.384+0000 7f2c2b7fe640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:01.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.384+0000 7f2c2b7fe640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-29T10:54:01.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.384+0000 7f2c42ffd640 10 monclient: get_auth_request con 0x7f2c3006dcf0 auth_method 0 2024-03-29T10:54:01.368 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.384+0000 7f2c2b7fe640 10 client.4297.objecter ms_handle_connect 0x7f2c3006dcf0 2024-03-29T10:54:01.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.392+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c3c04f4f0 con 0x7f2c4413f650 2024-03-29T10:54:01.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.392+0000 7f2c2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:01.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.392+0000 7f2c2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:01.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.713+0000 7f2c492b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:01.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.713+0000 7f2c492b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:01.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.713+0000 7f2c492b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:01.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.713+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c24005330 con 0x7f2c4413f650 2024-03-29T10:54:01.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.717+0000 7f2c2b7fe640 1 -- 172.21.15.155:0/1384033716 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f2c3c051420 con 0x7f2c4413f650 2024-03-29T10:54:01.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.717+0000 7f2c2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:01.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.717+0000 7f2c2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:01.699 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:01.699 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:01.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 10 librados: watch_flush enter 2024-03-29T10:54:01.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 10 librados: watch_flush exit 2024-03-29T10:54:01.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-29T10:54:01.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 10 client.4297.objecter successfully canceled tick 2024-03-29T10:54:01.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 >> v1:172.21.15.183:6816/1412921548 conn(0x7f2c3006dcf0 legacy=0x7f2c300701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:01.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 10 monclient: shutdown 2024-03-29T10:54:01.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:01.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 >> v1:172.21.15.183:6789/0 conn(0x7f2c4413f650 legacy=0x7f2c44071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:01.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 shutdown_connections 2024-03-29T10:54:01.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.721+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 >> 172.21.15.155:0/1384033716 conn(0x7f2c44066a00 msgr2=0x7f2c440b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:01.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.723+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 shutdown_connections 2024-03-29T10:54:01.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.723+0000 7f2c492b3640 1 -- 172.21.15.155:0/1384033716 wait complete. 2024-03-29T10:54:01.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:01.723+0000 7f2c492b3640 1 librados: shutdown 2024-03-29T10:54:01.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:08.530 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:08.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.719+0000 7faf89f2b640 10 monclient: get_monmap_and_config 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.719+0000 7faf89f2b640 10 monclient: build_initial_monmap 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.719+0000 7faf89f2b640 10 monclient: monmap: 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:08.719999+0000 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:08.719999+0000 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:08.702 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:08.703 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:08.703 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:08.703 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:08.703 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:08.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.720+0000 7faf89f2b640 1 Processor -- start 2024-03-29T10:54:08.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.720+0000 7faf89f2b640 1 -- start start 2024-03-29T10:54:08.703 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.720+0000 7faf89f2b640 10 monclient: init 2024-03-29T10:54:08.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.721+0000 7faf89f2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:08.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf89f2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:08.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf89f2b640 10 monclient(hunting): picked mon.noname-b con 0x7faf840ae250 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:08.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf89f2b640 10 monclient(hunting): picked mon.noname-c con 0x7faf8413f970 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:08.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf89f2b640 10 monclient(hunting): picked mon.noname-a con 0x7faf84142800 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:08.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf89f2b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840afe00 con 0x7faf84142800 2024-03-29T10:54:08.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf89f2b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf84142c10 con 0x7faf8413f970 2024-03-29T10:54:08.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf837fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7faf840ae250 0x7faf8413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37104/0 (socket says 172.21.15.155:37104) 2024-03-29T10:54:08.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.722+0000 7faf837fe640 1 -- 172.21.15.155:0/3173693755 learned_addr learned my addr 172.21.15.155:0/3173693755 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:08.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.725+0000 7faf89f2b640 1 -- 172.21.15.155:0/3173693755 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840afb20 con 0x7faf840ae250 2024-03-29T10:54:08.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.725+0000 7faf89f2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:08.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.725+0000 7faf89f2b640 10 monclient(hunting): authenticate will time out at 952.017753s 2024-03-29T10:54:08.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.727+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681142380 0 0) 0x7faf84142c10 con 0x7faf8413f970 2024-03-29T10:54:08.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.727+0000 7faf827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:08.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.727+0000 7faf827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:08.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.727+0000 7faf827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:08.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.727+0000 7faf827fc640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:54:08.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.728+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c003520 con 0x7faf8413f970 2024-03-29T10:54:08.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 898841746 0 0) 0x7faf6c003520 con 0x7faf8413f970 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 >> v1:172.21.15.155:6790/0 conn(0x7faf840ae250 legacy=0x7faf8413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 >> v1:172.21.15.155:6789/0 conn(0x7faf84142800 legacy=0x7faf8414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 monclient: found mon.noname-c 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf840aab80 con 0x7faf8413f970 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7faf74001690 con 0x7faf8413f970 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:08.711 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: dump: 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf740017f0 con 0x7faf8413f970 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 1 -- 172.21.15.155:0/3173693755 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7faf74001c40 con 0x7faf8413f970 2024-03-29T10:54:08.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf827fc640 10 monclient: dump: 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf89f2b640 5 monclient: authenticate success, global_id 4300 2024-03-29T10:54:08.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf89f2b640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:08.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.729+0000 7faf89f2b640 4 set_mon_vals no callback set 2024-03-29T10:54:08.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.730+0000 7faf89f2b640 10 monclient: shutdown 2024-03-29T10:54:08.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.730+0000 7faf89f2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:08.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.730+0000 7faf89f2b640 1 -- 172.21.15.155:0/3173693755 >> v1:172.21.15.183:6789/0 conn(0x7faf8413f970 legacy=0x7faf841420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:08.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.734+0000 7faf89f2b640 1 -- 172.21.15.155:0/3173693755 shutdown_connections 2024-03-29T10:54:08.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.734+0000 7faf89f2b640 1 -- 172.21.15.155:0/3173693755 >> 172.21.15.155:0/3173693755 conn(0x7faf840b35e0 msgr2=0x7faf840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:08.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.734+0000 7faf89f2b640 1 -- 172.21.15.155:0/3173693755 shutdown_connections 2024-03-29T10:54:08.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.734+0000 7faf89f2b640 1 -- 172.21.15.155:0/3173693755 wait complete. 2024-03-29T10:54:08.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.735+0000 7faf89f2b640 10 monclient: build_initial_monmap 2024-03-29T10:54:08.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.735+0000 7faf89f2b640 10 monclient: monmap: 2024-03-29T10:54:08.719 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.736+0000 7faf89f2b640 1 librados: starting msgr at 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.736+0000 7faf89f2b640 1 librados: starting objecter 2024-03-29T10:54:08.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.736+0000 7faf89f2b640 1 Processor -- start 2024-03-29T10:54:08.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.736+0000 7faf89f2b640 1 -- start start 2024-03-29T10:54:08.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.736+0000 7faf89f2b640 1 librados: setting wanted keys 2024-03-29T10:54:08.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.736+0000 7faf89f2b640 1 librados: calling monclient init 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.736+0000 7faf89f2b640 10 monclient: init 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.737+0000 7faf89f2b640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.737+0000 7faf89f2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.737+0000 7faf89f2b640 10 monclient(hunting): picked mon.noname-a con 0x7faf840ae250 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.737+0000 7faf89f2b640 10 monclient(hunting): picked mon.noname-c con 0x7faf8413f970 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf82ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7faf8413f970 0x7faf841317e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38632/0 (socket says 172.21.15.155:38632) 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf89f2b640 10 monclient(hunting): picked mon.noname-b con 0x7faf84142800 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf89f2b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840abae0 con 0x7faf840ae250 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf82ffd640 1 -- 172.21.15.155:0/2970322522 learned_addr learned my addr 172.21.15.155:0/2970322522 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840ad0b0 con 0x7faf84142800 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840ab5c0 con 0x7faf8413f970 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf89f2b640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:08.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf89f2b640 10 monclient(hunting): authenticate will time out at 952.033599s 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf6bfff640 10 client.?.objecter ms_handle_connect 0x7faf84142800 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf6bfff640 10 client.?.objecter ms_handle_connect 0x7faf8413f970 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.741+0000 7faf6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461738383 0 0) 0x7faf840ad0b0 con 0x7faf84142800 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:54:08.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf70004eb0 con 0x7faf84142800 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869179640 0 0) 0x7faf70004eb0 con 0x7faf84142800 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf840ad0b0 con 0x7faf84142800 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7faf7c001c30 con 0x7faf84142800 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): dump: 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.725 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69038141 0 0) 0x7faf840ab5c0 con 0x7faf8413f970 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf70004eb0 con 0x7faf8413f970 2024-03-29T10:54:08.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 438770397 0 0) 0x7faf840ad0b0 con 0x7faf84142800 2024-03-29T10:54:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 >> v1:172.21.15.155:6790/0 conn(0x7faf8413f970 legacy=0x7faf841317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.742+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 >> v1:172.21.15.155:6789/0 conn(0x7faf840ae250 legacy=0x7faf841d2d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 1 monclient: found mon.b 2024-03-29T10:54:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:08.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf840ab0a0 con 0x7faf84142800 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: _finish_auth 0 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 10 monclient: _renew_subs 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf840ab290 con 0x7faf84142800 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 10 monclient: _renew_subs 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf840abee0 con 0x7faf84142800 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf89f2b640 1 librados: init done 2024-03-29T10:54:08.728 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf7c003990 con 0x7faf84142800 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7faf7c003da0 con 0x7faf84142800 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf6bfff640 10 monclient: dump: 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:08.729 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:08.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.745+0000 7faf817fa640 4 set_mon_vals no callback set 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.747+0000 7faf89f2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.747+0000 7faf89f2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.747+0000 7faf89f2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.747+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf64005080 con 0x7faf84142800 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.748+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7faf7c003f00 con 0x7faf84142800 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.748+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7faf7c083c30 con 0x7faf84142800 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.748+0000 7faf6bfff640 10 client.4303.objecter ms_dispatch 0x7faf84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.748+0000 7faf6bfff640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.748+0000 7faf6bfff640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:08.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.749+0000 7faf6bfff640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T10:54:08.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.750+0000 7faf837fe640 10 monclient: get_auth_request con 0x7faf7006dd90 auth_method 0 2024-03-29T10:54:08.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.752+0000 7faf6bfff640 10 client.4303.objecter ms_handle_connect 0x7faf7006dd90 2024-03-29T10:54:08.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.755+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf7c04f510 con 0x7faf84142800 2024-03-29T10:54:08.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.755+0000 7faf6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:08.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:08.755+0000 7faf6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:09.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.106+0000 7faf89f2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:09.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.106+0000 7faf89f2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:09.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.106+0000 7faf89f2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:09.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.106+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf64005330 con 0x7faf84142800 2024-03-29T10:54:09.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.108+0000 7faf6bfff640 1 -- 172.21.15.155:0/2970322522 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7faf7c051440 con 0x7faf84142800 2024-03-29T10:54:09.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.108+0000 7faf6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:09.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.108+0000 7faf6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:09.090 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:09.090 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 10 librados: watch_flush enter 2024-03-29T10:54:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 10 librados: watch_flush exit 2024-03-29T10:54:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T10:54:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 10 client.4303.objecter successfully canceled tick 2024-03-29T10:54:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 >> v1:172.21.15.183:6816/1412921548 conn(0x7faf7006dd90 legacy=0x7faf70070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 10 monclient: shutdown 2024-03-29T10:54:09.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.112+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 >> v1:172.21.15.183:6789/0 conn(0x7faf84142800 legacy=0x7faf841de5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.113+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 shutdown_connections 2024-03-29T10:54:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.113+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 >> 172.21.15.155:0/2970322522 conn(0x7faf840b35e0 msgr2=0x7faf840b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.113+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 shutdown_connections 2024-03-29T10:54:09.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.113+0000 7faf89f2b640 1 -- 172.21.15.155:0/2970322522 wait complete. 2024-03-29T10:54:09.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:09.113+0000 7faf89f2b640 1 librados: shutdown 2024-03-29T10:54:09.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:15.917 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:16.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.081+0000 7f6807d3d640 10 monclient: get_monmap_and_config 2024-03-29T10:54:16.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.081+0000 7f6807d3d640 10 monclient: build_initial_monmap 2024-03-29T10:54:16.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.081+0000 7f6807d3d640 10 monclient: monmap: 2024-03-29T10:54:16.062 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:16.062 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:16.082232+0000 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:16.082232+0000 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:16.063 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:16.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.082+0000 7f6807d3d640 1 Processor -- start 2024-03-29T10:54:16.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.082+0000 7f6807d3d640 1 -- start start 2024-03-29T10:54:16.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.082+0000 7f6807d3d640 10 monclient: init 2024-03-29T10:54:16.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f68000b0860 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f6800072450 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f680013ee40 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000ad840 con 0x7f6800072450 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680013c830 con 0x7f68000b0860 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680013c9c0 con 0x7f680013ee40 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:16.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.083+0000 7f6807d3d640 10 monclient(hunting): authenticate will time out at 959.375668s 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6805ad9640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f68000b0860 0x7f68000acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:48076/0 (socket says 172.21.15.155:48076) 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6805ad9640 1 -- 172.21.15.155:0/1266234260 learned_addr learned my addr 172.21.15.155:0/1266234260 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935350113 0 0) 0x7f680013c9c0 con 0x7f680013ee40 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67ec003520 con 0x7f680013ee40 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852902627 0 0) 0x7f680013c830 con 0x7f68000b0860 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.084+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67ec003840 con 0x7f68000b0860 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2918353375 0 0) 0x7f67ec003840 con 0x7f68000b0860 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 >> v1:172.21.15.155:6790/0 conn(0x7f680013ee40 legacy=0x7f680013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 >> v1:172.21.15.155:6789/0 conn(0x7f6800072450 legacy=0x7f680013c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: _finish_hunting 0 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 monclient: found mon.noname-c 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680013f970 con 0x7f68000b0860 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: _finish_auth 0 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: _check_auth_tickets 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f67f8001630 con 0x7f68000b0860 2024-03-29T10:54:16.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:16.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: dump: 2024-03-29T10:54:16.073 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:16.073 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:16.073 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.073 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f8001880 con 0x7f68000b0860 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 1 -- 172.21.15.155:0/1266234260 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f67f8001c10 con 0x7f68000b0860 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.085+0000 7f6804ad7640 10 monclient: dump: 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.086+0000 7f6807d3d640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.086+0000 7f6807d3d640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:16.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.086+0000 7f6807d3d640 4 set_mon_vals no callback set 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.087+0000 7f6807d3d640 10 monclient: shutdown 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.087+0000 7f6807d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.087+0000 7f6807d3d640 1 -- 172.21.15.155:0/1266234260 >> v1:172.21.15.183:6789/0 conn(0x7f68000b0860 legacy=0x7f68000acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.087+0000 7f6807d3d640 1 -- 172.21.15.155:0/1266234260 shutdown_connections 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.087+0000 7f6807d3d640 1 -- 172.21.15.155:0/1266234260 >> 172.21.15.155:0/1266234260 conn(0x7f680006b660 msgr2=0x7f680006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.087+0000 7f6807d3d640 1 -- 172.21.15.155:0/1266234260 shutdown_connections 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.087+0000 7f6807d3d640 1 -- 172.21.15.155:0/1266234260 wait complete. 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.088+0000 7f6807d3d640 10 monclient: build_initial_monmap 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.088+0000 7f6807d3d640 10 monclient: monmap: 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:16.076 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.089+0000 7f6807d3d640 1 librados: starting msgr at 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.089+0000 7f6807d3d640 1 librados: starting objecter 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.089+0000 7f6807d3d640 1 Processor -- start 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.089+0000 7f6807d3d640 1 -- start start 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.089+0000 7f6807d3d640 1 librados: setting wanted keys 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.089+0000 7f6807d3d640 1 librados: calling monclient init 2024-03-29T10:54:16.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.089+0000 7f6807d3d640 10 monclient: init 2024-03-29T10:54:16.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:16.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:16.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f6800072450 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:16.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f68000b0860 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:16.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f680013ee40 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:16.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6805ad9640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6800072450 0x7f68001df1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:37010/0 (socket says 172.21.15.155:37010) 2024-03-29T10:54:16.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6805ad9640 1 -- 172.21.15.155:0/1481830141 learned_addr learned my addr 172.21.15.155:0/1481830141 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68001ded10 con 0x7f6800072450 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800070390 con 0x7f68000b0860 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6800070670 con 0x7f680013ee40 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f6807d3d640 10 monclient(hunting): authenticate will time out at 959.382448s 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f67f67fc640 10 client.?.objecter ms_handle_connect 0x7f6800072450 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f67f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f67f67fc640 10 client.?.objecter ms_handle_connect 0x7f680013ee40 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.090+0000 7f67f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363636938 0 0) 0x7f68001ded10 con 0x7f6800072450 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e4004eb0 con 0x7f6800072450 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 client.?.objecter ms_handle_connect 0x7f68000b0860 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160818970 0 0) 0x7f6800070670 con 0x7f680013ee40 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68001ded10 con 0x7f680013ee40 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2123389419 0 0) 0x7f67e4004eb0 con 0x7f6800072450 2024-03-29T10:54:16.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6800070670 con 0x7f6800072450 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f67f8001c10 con 0x7f6800072450 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): dump: 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:16.087 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.091+0000 7f67f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286194598 0 0) 0x7f6800070390 con 0x7f68000b0860 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e4004eb0 con 0x7f68000b0860 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3712368221 0 0) 0x7f6800070670 con 0x7f6800072450 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 >> v1:172.21.15.155:6790/0 conn(0x7f680013ee40 legacy=0x7f6800134ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 >> v1:172.21.15.183:6789/0 conn(0x7f68000b0860 legacy=0x7f68000710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 1 monclient: found mon.a 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:16.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000b26a0 con 0x7f6800072450 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f67f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 5 monclient: authenticate success, global_id 4308 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 10 monclient: _renew_subs 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68001df900 con 0x7f6800072450 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 10 monclient: _renew_subs 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68000719d0 con 0x7f6800072450 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.092+0000 7f6807d3d640 1 librados: init done 2024-03-29T10:54:16.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.093+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f8001190 con 0x7f6800072450 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.093+0000 7f67f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.093+0000 7f67f7fff640 4 set_mon_vals no callback set 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.094+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f67f8004c70 con 0x7f6800072450 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.094+0000 7f67f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.094+0000 7f67f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.094+0000 7f67f67fc640 10 monclient: dump: 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:16.090 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.094+0000 7f6807d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.095+0000 7f6807d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.095+0000 7f6807d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.095+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e8005080 con 0x7f6800072450 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.096+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f67f8004dd0 con 0x7f6800072450 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.096+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f67f8084c30 con 0x7f6800072450 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.096+0000 7f67f67fc640 10 client.4308.objecter ms_dispatch 0x7f6800000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.096+0000 7f67f67fc640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.096+0000 7f67f67fc640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.096+0000 7f67f67fc640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T10:54:16.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.097+0000 7f68052d8640 10 monclient: get_auth_request con 0x7f67e406ddf0 auth_method 0 2024-03-29T10:54:16.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.097+0000 7f67f67fc640 10 client.4308.objecter ms_handle_connect 0x7f67e406ddf0 2024-03-29T10:54:16.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.107+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67f8050340 con 0x7f6800072450 2024-03-29T10:54:16.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.107+0000 7f67f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:16.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.107+0000 7f67f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:16.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.479+0000 7f6807d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:16.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.479+0000 7f6807d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:16.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.479+0000 7f6807d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:16.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.479+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67e8005330 con 0x7f6800072450 2024-03-29T10:54:16.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.480+0000 7f67f67fc640 1 -- 172.21.15.155:0/1481830141 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f67f8052440 con 0x7f6800072450 2024-03-29T10:54:16.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.480+0000 7f67f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:16.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.480+0000 7f67f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:16.461 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:16.461 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.483+0000 7f6807d3d640 10 librados: watch_flush enter 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 10 librados: watch_flush exit 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 10 client.4308.objecter successfully canceled tick 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 >> v1:172.21.15.183:6816/1412921548 conn(0x7f67e406ddf0 legacy=0x7f67e40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 10 monclient: shutdown 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 >> v1:172.21.15.155:6789/0 conn(0x7f6800072450 legacy=0x7f68001df1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 shutdown_connections 2024-03-29T10:54:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.484+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 >> 172.21.15.155:0/1481830141 conn(0x7f680006b660 msgr2=0x7f68000a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.485+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 shutdown_connections 2024-03-29T10:54:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.485+0000 7f6807d3d640 1 -- 172.21.15.155:0/1481830141 wait complete. 2024-03-29T10:54:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:16.485+0000 7f6807d3d640 1 librados: shutdown 2024-03-29T10:54:16.481 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:23.289 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:23.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.440+0000 7fd253255640 10 monclient: get_monmap_and_config 2024-03-29T10:54:23.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.440+0000 7fd253255640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.440+0000 7fd253255640 10 monclient: monmap: 2024-03-29T10:54:23.421 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:23.421 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:23.441461+0000 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:23.441461+0000 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:23.422 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:23.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.441+0000 7fd253255640 1 Processor -- start 2024-03-29T10:54:23.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.441+0000 7fd253255640 1 -- start start 2024-03-29T10:54:23.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.441+0000 7fd253255640 10 monclient: init 2024-03-29T10:54:23.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.423 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 10 monclient(hunting): picked mon.noname-b con 0x7fd24c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 10 monclient(hunting): picked mon.noname-a con 0x7fd24c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 10 monclient(hunting): picked mon.noname-c con 0x7fd24c0a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c0aaef0 con 0x7fd24c13f650 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c145bd0 con 0x7fd24c0a9d70 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd250ff1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd24c1457f0 0x7fd24c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43518/0 (socket says 172.21.15.155:43518) 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd250ff1640 1 -- 172.21.15.155:0/3587919235 learned_addr learned my addr 172.21.15.155:0/3587919235 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 1 -- 172.21.15.155:0/3587919235 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c145eb0 con 0x7fd24c1457f0 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.442+0000 7fd253255640 10 monclient(hunting): authenticate will time out at 966.734876s 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502764820 0 0) 0x7fd24c145eb0 con 0x7fd24c1457f0 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.424 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:54:23.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c003520 con 0x7fd24c1457f0 2024-03-29T10:54:23.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837708491 0 0) 0x7fd23c003520 con 0x7fd24c1457f0 2024-03-29T10:54:23.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 >> v1:172.21.15.183:6789/0 conn(0x7fd24c0a9d70 legacy=0x7fd24c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:23.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.443+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 >> v1:172.21.15.155:6789/0 conn(0x7fd24c13f650 legacy=0x7fd24c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c146340 con 0x7fd24c1457f0 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd240001690 con 0x7fd24c1457f0 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd24b7fe640 10 monclient: dump: 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:23.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd253255640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:54:23.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.446+0000 7fd253255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:23.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd240001920 con 0x7fd24c1457f0 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd24b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd24b7fe640 1 -- 172.21.15.155:0/3587919235 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd240001cb0 con 0x7fd24c1457f0 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd24b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd24b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd24b7fe640 10 monclient: dump: 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:23.431 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd253255640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.447+0000 7fd253255640 4 set_mon_vals no callback set 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.448+0000 7fd253255640 10 monclient: shutdown 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.448+0000 7fd253255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.448+0000 7fd253255640 1 -- 172.21.15.155:0/3587919235 >> v1:172.21.15.155:6790/0 conn(0x7fd24c1457f0 legacy=0x7fd24c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.448+0000 7fd253255640 1 -- 172.21.15.155:0/3587919235 shutdown_connections 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.448+0000 7fd253255640 1 -- 172.21.15.155:0/3587919235 >> 172.21.15.155:0/3587919235 conn(0x7fd24c066a00 msgr2=0x7fd24c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.448+0000 7fd253255640 1 -- 172.21.15.155:0/3587919235 shutdown_connections 2024-03-29T10:54:23.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.448+0000 7fd253255640 1 -- 172.21.15.155:0/3587919235 wait complete. 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.449+0000 7fd253255640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.449+0000 7fd253255640 10 monclient: monmap: 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.450+0000 7fd253255640 1 librados: starting msgr at 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.450+0000 7fd253255640 1 librados: starting objecter 2024-03-29T10:54:23.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.450+0000 7fd253255640 1 Processor -- start 2024-03-29T10:54:23.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.450+0000 7fd253255640 1 -- start start 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.450+0000 7fd253255640 1 librados: setting wanted keys 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.450+0000 7fd253255640 1 librados: calling monclient init 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.450+0000 7fd253255640 10 monclient: init 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.451+0000 7fd253255640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.451+0000 7fd253255640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.451+0000 7fd253255640 10 monclient(hunting): picked mon.noname-c con 0x7fd24c0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.451+0000 7fd250ff1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fd24c0a9d70 0x7fd24c13a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43954/0 (socket says 172.21.15.155:43954) 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.451+0000 7fd250ff1640 1 -- 172.21.15.155:0/2490142392 learned_addr learned my addr 172.21.15.155:0/2490142392 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.451+0000 7fd253255640 10 monclient(hunting): picked mon.noname-b con 0x7fd24c13f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 client.?.objecter ms_handle_connect 0x7fd24c0a9d70 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd253255640 10 monclient(hunting): picked mon.noname-a con 0x7fd24c1457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:23.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c13a3d0 con 0x7fd24c1457f0 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c139de0 con 0x7fd24c13f650 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c139f70 con 0x7fd24c0a9d70 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd253255640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd253255640 10 monclient(hunting): authenticate will time out at 966.744508s 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683216738 0 0) 0x7fd24c139f70 con 0x7fd24c0a9d70 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230004eb0 con 0x7fd24c0a9d70 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1853995931 0 0) 0x7fd230004eb0 con 0x7fd24c0a9d70 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd24c139f70 con 0x7fd24c0a9d70 2024-03-29T10:54:23.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd240001650 con 0x7fd24c0a9d70 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.452+0000 7fd2497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.453+0000 7fd2497fa640 10 monclient(hunting): dump: 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:23.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.453+0000 7fd2497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.453+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 962611481 0 0) 0x7fd24c139f70 con 0x7fd24c0a9d70 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.453+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 >> v1:172.21.15.183:6789/0 conn(0x7fd24c13f650 legacy=0x7fd24c1da5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.453+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 >> v1:172.21.15.155:6789/0 conn(0x7fd24c1457f0 legacy=0x7fd24c13c260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 1 monclient: found mon.c 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c142380 con 0x7fd24c0a9d70 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd2497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd253255640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd253255640 10 monclient: _renew_subs 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd253255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:23.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.454+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24c1425d0 con 0x7fd24c0a9d70 2024-03-29T10:54:23.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd253255640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:23.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd253255640 10 monclient: _renew_subs 2024-03-29T10:54:23.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd253255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:23.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24c135f20 con 0x7fd24c0a9d70 2024-03-29T10:54:23.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd253255640 1 librados: init done 2024-03-29T10:54:23.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd240004a90 con 0x7fd24c0a9d70 2024-03-29T10:54:23.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd2497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd240004e60 con 0x7fd24c0a9d70 2024-03-29T10:54:23.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd2497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd2497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:23.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.455+0000 7fd2497fa640 10 monclient: dump: 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.456+0000 7fd24affd640 4 set_mon_vals no callback set 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.456+0000 7fd253255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.456+0000 7fd253255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.456+0000 7fd253255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.456+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c005080 con 0x7fd24c0a9d70 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.458+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fd240004fc0 con 0x7fd24c0a9d70 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.458+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fd240084c30 con 0x7fd24c0a9d70 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.458+0000 7fd2497fa640 10 client.4331.objecter ms_dispatch 0x7fd24c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.458+0000 7fd2497fa640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.458+0000 7fd2497fa640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.459+0000 7fd2497fa640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.459+0000 7fd24bfff640 10 monclient: get_auth_request con 0x7fd23006dbd0 auth_method 0 2024-03-29T10:54:23.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.459+0000 7fd2497fa640 10 client.4331.objecter ms_handle_connect 0x7fd23006dbd0 2024-03-29T10:54:23.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.465+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd240050510 con 0x7fd24c0a9d70 2024-03-29T10:54:23.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.465+0000 7fd2497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.465+0000 7fd2497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:23.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.772+0000 7fd253255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:23.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.772+0000 7fd253255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:23.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.772+0000 7fd253255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:23.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.772+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd22c005330 con 0x7fd24c0a9d70 2024-03-29T10:54:23.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.774+0000 7fd2497fa640 1 -- 172.21.15.155:0/2490142392 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fd240052440 con 0x7fd24c0a9d70 2024-03-29T10:54:23.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.774+0000 7fd2497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:23.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.774+0000 7fd2497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:23.755 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:23.755 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:23.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 10 librados: watch_flush enter 2024-03-29T10:54:23.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 10 librados: watch_flush exit 2024-03-29T10:54:23.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T10:54:23.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 10 client.4331.objecter successfully canceled tick 2024-03-29T10:54:23.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 >> v1:172.21.15.183:6816/1412921548 conn(0x7fd23006dbd0 legacy=0x7fd230070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 10 monclient: shutdown 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 >> v1:172.21.15.155:6790/0 conn(0x7fd24c0a9d70 legacy=0x7fd24c13a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 shutdown_connections 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 >> 172.21.15.155:0/2490142392 conn(0x7fd24c066a00 msgr2=0x7fd24c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 shutdown_connections 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 1 -- 172.21.15.155:0/2490142392 wait complete. 2024-03-29T10:54:23.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:23.777+0000 7fd253255640 1 librados: shutdown 2024-03-29T10:54:23.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:30.579 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:30.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.744+0000 7f706afe8640 10 monclient: get_monmap_and_config 2024-03-29T10:54:30.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.745+0000 7f706afe8640 10 monclient: build_initial_monmap 2024-03-29T10:54:30.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.745+0000 7f706afe8640 10 monclient: monmap: 2024-03-29T10:54:30.726 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:30.726 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:30.726 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:30.745687+0000 2024-03-29T10:54:30.726 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:30.745687+0000 2024-03-29T10:54:30.727 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:30.727 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.727 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:30.727 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:30.727 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:30.727 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.747+0000 7f706afe8640 1 Processor -- start 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.747+0000 7f706afe8640 1 -- start start 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.747+0000 7f706afe8640 10 monclient: init 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 10 monclient(hunting): picked mon.noname-b con 0x7f70640a9930 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 10 monclient(hunting): picked mon.noname-c con 0x7f7064149dc0 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 10 monclient(hunting): picked mon.noname-a con 0x7f706414ff40 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70640aae60 con 0x7f706414ff40 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70640b4310 con 0x7f7064149dc0 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70640afdd0 con 0x7f70640a9930 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.748+0000 7f706afe8640 10 monclient(hunting): authenticate will time out at 974.040228s 2024-03-29T10:54:30.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.750+0000 7f7063fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7064149dc0 0x7f706414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:38870/0 (socket says 172.21.15.155:38870) 2024-03-29T10:54:30.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.750+0000 7f7063fff640 1 -- 172.21.15.155:0/768895670 learned_addr learned my addr 172.21.15.155:0/768895670 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243333529 0 0) 0x7f70640b4310 con 0x7f7064149dc0 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f704c003520 con 0x7f7064149dc0 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013781272 0 0) 0x7f70640aae60 con 0x7f706414ff40 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:30.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:54:30.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.753+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70640b4310 con 0x7f706414ff40 2024-03-29T10:54:30.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 826757711 0 0) 0x7f70640b4310 con 0x7f706414ff40 2024-03-29T10:54:30.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 >> v1:172.21.15.155:6790/0 conn(0x7f70640a9930 legacy=0x7f7064143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:30.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 >> v1:172.21.15.183:6789/0 conn(0x7f7064149dc0 legacy=0x7f706414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:30.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 1 monclient: found mon.noname-a 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706414a950 con 0x7f706414ff40 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f706afe8640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f706afe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7054001520 con 0x7f706414ff40 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:30.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: dump: 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f706afe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7054001740 con 0x7f706414ff40 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f70637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f706afe8640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:30.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.754+0000 7f706afe8640 4 set_mon_vals no callback set 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f70637fe640 1 -- 172.21.15.155:0/768895670 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7054001ad0 con 0x7f706414ff40 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f70637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f70637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f70637fe640 10 monclient: dump: 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:30.741 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f706afe8640 10 monclient: shutdown 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f706afe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f706afe8640 1 -- 172.21.15.155:0/768895670 >> v1:172.21.15.155:6789/0 conn(0x7f706414ff40 legacy=0x7f7064152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f706afe8640 1 -- 172.21.15.155:0/768895670 shutdown_connections 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.755+0000 7f706afe8640 1 -- 172.21.15.155:0/768895670 >> 172.21.15.155:0/768895670 conn(0x7f70640b35e0 msgr2=0x7f70640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.756+0000 7f706afe8640 1 -- 172.21.15.155:0/768895670 shutdown_connections 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.756+0000 7f706afe8640 1 -- 172.21.15.155:0/768895670 wait complete. 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.756+0000 7f706afe8640 10 monclient: build_initial_monmap 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.756+0000 7f706afe8640 10 monclient: monmap: 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:30.742 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.757+0000 7f706afe8640 1 librados: starting msgr at 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.757+0000 7f706afe8640 1 librados: starting objecter 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.757+0000 7f706afe8640 1 Processor -- start 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.757+0000 7f706afe8640 1 -- start start 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.757+0000 7f706afe8640 1 librados: setting wanted keys 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.757+0000 7f706afe8640 1 librados: calling monclient init 2024-03-29T10:54:30.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.757+0000 7f706afe8640 10 monclient: init 2024-03-29T10:54:30.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.758+0000 7f706afe8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:30.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.758+0000 7f706afe8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:30.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.758+0000 7f706afe8640 10 monclient(hunting): picked mon.noname-b con 0x7f70640a9930 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.760+0000 7f706afe8640 10 monclient(hunting): picked mon.noname-c con 0x7f7064149dc0 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f706afe8640 10 monclient(hunting): picked mon.noname-a con 0x7f706414ff40 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f706afe8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70641de810 con 0x7f706414ff40 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f706afe8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70641dead0 con 0x7f70640a9930 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f706afe8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706406e7a0 con 0x7f7064149dc0 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f706afe8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f706afe8640 10 monclient(hunting): authenticate will time out at 974.053008s 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f7068d84640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f70640a9930 0x7f706406eb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:39202/0 (socket says 172.21.15.155:39202) 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.761+0000 7f7068d84640 1 -- 172.21.15.155:0/1999487701 learned_addr learned my addr 172.21.15.155:0/1999487701 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.763+0000 7f70617fa640 10 client.?.objecter ms_handle_connect 0x7f70640a9930 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 client.?.objecter ms_handle_connect 0x7f706414ff40 2024-03-29T10:54:30.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 client.?.objecter ms_handle_connect 0x7f7064149dc0 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194446560 0 0) 0x7f70641de810 con 0x7f706414ff40 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7050004eb0 con 0x7f706414ff40 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925514970 0 0) 0x7f70641dead0 con 0x7f70640a9930 2024-03-29T10:54:30.748 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.764+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70641de810 con 0x7f70640a9930 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516468052 0 0) 0x7f706406e7a0 con 0x7f7064149dc0 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70641dead0 con 0x7f7064149dc0 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015317843 0 0) 0x7f7050004eb0 con 0x7f706414ff40 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f706406e7a0 con 0x7f706414ff40 2024-03-29T10:54:30.749 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7054001a50 con 0x7f706414ff40 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): dump: 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773315728 0 0) 0x7f70641de810 con 0x7f70640a9930 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7050004eb0 con 0x7f70640a9930 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f705c001b60 con 0x7f70640a9930 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 10 monclient(hunting): dump: 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.751 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.765+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264433813 0 0) 0x7f70641dead0 con 0x7f7064149dc0 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70641de810 con 0x7f7064149dc0 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f70580018b0 con 0x7f7064149dc0 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient(hunting): dump: 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.752 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4085374443 0 0) 0x7f70641de810 con 0x7f7064149dc0 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 >> v1:172.21.15.183:6789/0 conn(0x7f70640a9930 legacy=0x7f706406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 >> v1:172.21.15.155:6789/0 conn(0x7f706414ff40 legacy=0x7f70641e5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 monclient: found mon.c 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706406e930 con 0x7f7064149dc0 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:30.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7058001a10 con 0x7f7064149dc0 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.766+0000 7f70627fc640 4 set_mon_vals no callback set 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7058004980 con 0x7f7064149dc0 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f70617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f70617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f70617fa640 10 monclient: dump: 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:30.754 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:30.755 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:30.755 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:30.755 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:30.755 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:30.755 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:30.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:54:30.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 10 monclient: _renew_subs 2024-03-29T10:54:30.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:30.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706406fb50 con 0x7f7064149dc0 2024-03-29T10:54:30.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:30.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 10 monclient: _renew_subs 2024-03-29T10:54:30.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70641e72a0 con 0x7f7064149dc0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.767+0000 7f706afe8640 1 librados: init done 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.769+0000 7f704affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.769+0000 7f704affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.769+0000 7f704affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.769+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706414fd70 con 0x7f7064149dc0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.770+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f705801a950 con 0x7f7064149dc0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.770+0000 7f7068d84640 10 monclient: get_auth_request con 0x7f705006e160 auth_method 0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.771+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f70580849b0 con 0x7f7064149dc0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.771+0000 7f70617fa640 10 client.4334.objecter ms_dispatch 0x7f7064000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.771+0000 7f70617fa640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.771+0000 7f70617fa640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.771+0000 7f70617fa640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T10:54:30.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.771+0000 7f70617fa640 10 client.4334.objecter ms_handle_connect 0x7f705006e160 2024-03-29T10:54:30.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.779+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7058050290 con 0x7f7064149dc0 2024-03-29T10:54:30.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.779+0000 7f70617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:30.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:30.779+0000 7f70617fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:31.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.104+0000 7f704affd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:31.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.104+0000 7f704affd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:31.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.104+0000 7f704affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:31.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.104+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f706406fce0 con 0x7f7064149dc0 2024-03-29T10:54:31.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.106+0000 7f70617fa640 1 -- 172.21.15.155:0/1999487701 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f70580521c0 con 0x7f7064149dc0 2024-03-29T10:54:31.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.106+0000 7f70617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:31.087 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.106+0000 7f70617fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:31.087 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:31.087 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.109+0000 7f704affd640 10 librados: watch_flush enter 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.109+0000 7f704affd640 10 librados: watch_flush exit 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.109+0000 7f704affd640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.109+0000 7f704affd640 10 client.4334.objecter successfully canceled tick 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 >> v1:172.21.15.183:6816/1412921548 conn(0x7f705006e160 legacy=0x7f7050070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 10 monclient: shutdown 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 >> v1:172.21.15.155:6790/0 conn(0x7f7064149dc0 legacy=0x7f70641deff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 shutdown_connections 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 >> 172.21.15.155:0/1999487701 conn(0x7f70640b35e0 msgr2=0x7f70640b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 shutdown_connections 2024-03-29T10:54:31.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 1 -- 172.21.15.155:0/1999487701 wait complete. 2024-03-29T10:54:31.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:31.110+0000 7f704affd640 1 librados: shutdown 2024-03-29T10:54:31.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:37.918 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:38.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.089+0000 7f15d31f6640 10 monclient: get_monmap_and_config 2024-03-29T10:54:38.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.089+0000 7f15d31f6640 10 monclient: build_initial_monmap 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.090+0000 7f15d31f6640 10 monclient: monmap: 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:38.091649+0000 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:38.091649+0000 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:38.072 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:38.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.091+0000 7f15d31f6640 1 Processor -- start 2024-03-29T10:54:38.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.091+0000 7f15d31f6640 1 -- start start 2024-03-29T10:54:38.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.091+0000 7f15d31f6640 10 monclient: init 2024-03-29T10:54:38.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:38.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:38.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 10 monclient(hunting): picked mon.noname-c con 0x7f15c4006da0 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:38.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 10 monclient(hunting): picked mon.noname-a con 0x7f15c40cc330 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:38.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 10 monclient(hunting): picked mon.noname-b con 0x7f15c40c9450 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:38.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c4008490 con 0x7f15c40cc330 2024-03-29T10:54:38.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c402be50 con 0x7f15c4006da0 2024-03-29T10:54:38.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c402c130 con 0x7f15c40c9450 2024-03-29T10:54:38.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:38.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.092+0000 7f15d31f6640 10 monclient(hunting): authenticate will time out at 981.385588s 2024-03-29T10:54:38.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.096+0000 7f15d0f92640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f15c4006da0 0x7f15c40cbc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:38029/0 (socket says 172.21.15.155:38029) 2024-03-29T10:54:38.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.096+0000 7f15d0f92640 1 -- 172.21.15.155:0/2763886411 learned_addr learned my addr 172.21.15.155:0/2763886411 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:38.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108053962 0 0) 0x7f15c402be50 con 0x7f15c4006da0 2024-03-29T10:54:38.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:38.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:38.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:38.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:54:38.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15bc003520 con 0x7f15c4006da0 2024-03-29T10:54:38.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2970229516 0 0) 0x7f15bc003520 con 0x7f15c4006da0 2024-03-29T10:54:38.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 >> v1:172.21.15.155:6790/0 conn(0x7f15c40c9450 legacy=0x7f15c40d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:38.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.098+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 >> v1:172.21.15.155:6789/0 conn(0x7f15c40cc330 legacy=0x7f15c40c8d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:38.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.099+0000 7f15cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:38.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.099+0000 7f15cb7fe640 1 monclient: found mon.noname-c 2024-03-29T10:54:38.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.099+0000 7f15cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:38.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.099+0000 7f15cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.099+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c402c410 con 0x7f15c4006da0 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15cb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f15c0001670 con 0x7f15c4006da0 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15cb7fe640 10 monclient: dump: 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:38.083 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15d31f6640 5 monclient: authenticate success, global_id 4318 2024-03-29T10:54:38.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.100+0000 7f15d31f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.101+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c0001890 con 0x7f15c4006da0 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.101+0000 7f15cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.101+0000 7f15d31f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.101+0000 7f15d31f6640 4 set_mon_vals no callback set 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.102+0000 7f15d31f6640 10 monclient: shutdown 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.102+0000 7f15d31f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.102+0000 7f15d31f6640 1 -- 172.21.15.155:0/2763886411 >> v1:172.21.15.183:6789/0 conn(0x7f15c4006da0 legacy=0x7f15c40cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.102+0000 7f15cb7fe640 1 -- 172.21.15.155:0/2763886411 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f15c0001c60 con 0x7f15c4006da0 2024-03-29T10:54:38.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.102+0000 7f15cb7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:54:38.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.105+0000 7f15d31f6640 1 -- 172.21.15.155:0/2763886411 shutdown_connections 2024-03-29T10:54:38.086 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.105+0000 7f15d31f6640 1 -- 172.21.15.155:0/2763886411 >> 172.21.15.155:0/2763886411 conn(0x7f15c4005c20 msgr2=0x7f15c4006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:38.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.105+0000 7f15d31f6640 1 -- 172.21.15.155:0/2763886411 shutdown_connections 2024-03-29T10:54:38.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.108+0000 7f15d31f6640 1 -- 172.21.15.155:0/2763886411 wait complete. 2024-03-29T10:54:38.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.109+0000 7f15d31f6640 10 monclient: build_initial_monmap 2024-03-29T10:54:38.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.109+0000 7f15d31f6640 10 monclient: monmap: 2024-03-29T10:54:38.090 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:38.090 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:38.091 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:38.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.109+0000 7f15d31f6640 1 librados: starting msgr at 2024-03-29T10:54:38.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.109+0000 7f15d31f6640 1 librados: starting objecter 2024-03-29T10:54:38.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.109+0000 7f15d31f6640 1 Processor -- start 2024-03-29T10:54:38.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.112+0000 7f15d31f6640 1 -- start start 2024-03-29T10:54:38.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.112+0000 7f15d31f6640 1 librados: setting wanted keys 2024-03-29T10:54:38.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.112+0000 7f15d31f6640 1 librados: calling monclient init 2024-03-29T10:54:38.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.112+0000 7f15d31f6640 10 monclient: init 2024-03-29T10:54:38.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 10 monclient(hunting): picked mon.noname-b con 0x7f15c4006da0 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 10 monclient(hunting): picked mon.noname-a con 0x7f15c40c9450 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 10 monclient(hunting): picked mon.noname-c con 0x7f15c40cc330 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c4034110 con 0x7f15c40c9450 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c40343f0 con 0x7f15c4006da0 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c4160b60 con 0x7f15c40cc330 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:38.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.113+0000 7f15d31f6640 10 monclient(hunting): authenticate will time out at 981.406510s 2024-03-29T10:54:38.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15d0f92640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f15c4006da0 0x7f15c4032d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:38197/0 (socket says 172.21.15.155:38197) 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15d0f92640 1 -- 172.21.15.155:0/3875975578 learned_addr learned my addr 172.21.15.155:0/3875975578 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 client.?.objecter ms_handle_connect 0x7f15c4006da0 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579748594 0 0) 0x7f15c40343f0 con 0x7f15c4006da0 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b0004eb0 con 0x7f15c4006da0 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613281414 0 0) 0x7f15b0004eb0 con 0x7f15c4006da0 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c40343f0 con 0x7f15c4006da0 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f15c0001ca0 con 0x7f15c4006da0 2024-03-29T10:54:38.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): dump: 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:38.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.117+0000 7f15c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3224259006 0 0) 0x7f15c40343f0 con 0x7f15c4006da0 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 >> v1:172.21.15.155:6790/0 conn(0x7f15c40cc330 legacy=0x7f15c40c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 >> v1:172.21.15.155:6789/0 conn(0x7f15c40c9450 legacy=0x7f15c4161080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 1 monclient: found mon.b 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c402f3f0 con 0x7f15c4006da0 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c0004b00 con 0x7f15c4006da0 2024-03-29T10:54:38.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f15c0004ed0 con 0x7f15c4006da0 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.118+0000 7f15c97fa640 10 monclient: dump: 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:38.106 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 5 monclient: authenticate success, global_id 4321 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 10 monclient: _renew_subs 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c4161790 con 0x7f15c4006da0 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 10 monclient: _renew_subs 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:38.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c40c5360 con 0x7f15c4006da0 2024-03-29T10:54:38.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15d31f6640 1 librados: init done 2024-03-29T10:54:38.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.119+0000 7f15caffd640 4 set_mon_vals no callback set 2024-03-29T10:54:38.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.123+0000 7f15d31f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:38.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.123+0000 7f15d31f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:38.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.123+0000 7f15d31f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:38.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.123+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ac005080 con 0x7f15c4006da0 2024-03-29T10:54:38.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.124+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f15c0005030 con 0x7f15c4006da0 2024-03-29T10:54:38.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.127+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f15c0084d80 con 0x7f15c4006da0 2024-03-29T10:54:38.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.127+0000 7f15c97fa640 10 client.4321.objecter ms_dispatch 0x7f15cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:38.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.127+0000 7f15c97fa640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:38.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.127+0000 7f15c97fa640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:38.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.127+0000 7f15c97fa640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T10:54:38.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.128+0000 7f15cbfff640 10 monclient: get_auth_request con 0x7f15b006db10 auth_method 0 2024-03-29T10:54:38.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.130+0000 7f15c97fa640 10 client.4321.objecter ms_handle_connect 0x7f15b006db10 2024-03-29T10:54:38.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.134+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15c00506c0 con 0x7f15c4006da0 2024-03-29T10:54:38.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.135+0000 7f15c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:38.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.135+0000 7f15c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:38.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.458+0000 7f15d31f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:38.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.458+0000 7f15d31f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:38.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.458+0000 7f15d31f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:54:38.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.458+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15ac005330 con 0x7f15c4006da0 2024-03-29T10:54:38.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.460+0000 7f15c97fa640 1 -- 172.21.15.155:0/3875975578 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f15c0084430 con 0x7f15c4006da0 2024-03-29T10:54:38.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.460+0000 7f15c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:38.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.460+0000 7f15c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:38.442 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:38.442 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:38.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 10 librados: watch_flush enter 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 10 librados: watch_flush exit 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 10 client.4321.objecter successfully canceled tick 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 >> v1:172.21.15.183:6816/1412921548 conn(0x7f15b006db10 legacy=0x7f15b006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 10 monclient: shutdown 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 >> v1:172.21.15.183:6789/0 conn(0x7f15c4006da0 legacy=0x7f15c4032d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 shutdown_connections 2024-03-29T10:54:38.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.464+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 >> 172.21.15.155:0/3875975578 conn(0x7f15c4005c20 msgr2=0x7f15c40c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:38.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.465+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 shutdown_connections 2024-03-29T10:54:38.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.466+0000 7f15d31f6640 1 -- 172.21.15.155:0/3875975578 wait complete. 2024-03-29T10:54:38.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:38.466+0000 7f15d31f6640 1 librados: shutdown 2024-03-29T10:54:38.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:45.276 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:45.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.450+0000 7f10c379d640 10 monclient: get_monmap_and_config 2024-03-29T10:54:45.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.450+0000 7f10c379d640 10 monclient: build_initial_monmap 2024-03-29T10:54:45.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.450+0000 7f10c379d640 10 monclient: monmap: 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:45.452516+0000 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:45.452516+0000 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:45.434 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:45.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.453+0000 7f10c379d640 1 Processor -- start 2024-03-29T10:54:45.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.453+0000 7f10c379d640 1 -- start start 2024-03-29T10:54:45.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.453+0000 7f10c379d640 10 monclient: init 2024-03-29T10:54:45.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:45.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:45.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 10 monclient(hunting): picked mon.noname-a con 0x7f10bc0ae250 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:45.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 10 monclient(hunting): picked mon.noname-c con 0x7f10bc13f970 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:45.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 10 monclient(hunting): picked mon.noname-b con 0x7f10bc142800 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:45.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc0afe00 con 0x7f10bc0ae250 2024-03-29T10:54:45.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc142c10 con 0x7f10bc13f970 2024-03-29T10:54:45.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc0afb20 con 0x7f10bc142800 2024-03-29T10:54:45.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:45.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.454+0000 7f10c379d640 10 monclient(hunting): authenticate will time out at 988.747744s 2024-03-29T10:54:45.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10c1d3a640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f10bc142800 0x7f10bc14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50483/0 (socket says 172.21.15.155:50483) 2024-03-29T10:54:45.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10c1d3a640 1 -- 172.21.15.155:0/158960789 learned_addr learned my addr 172.21.15.155:0/158960789 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761502667 0 0) 0x7f10bc0afb20 con 0x7f10bc142800 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a8003520 con 0x7f10bc142800 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1171691199 0 0) 0x7f10a8003520 con 0x7f10bc142800 2024-03-29T10:54:45.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.459+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 >> v1:172.21.15.183:6789/0 conn(0x7f10bc13f970 legacy=0x7f10bc1420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:45.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 >> v1:172.21.15.155:6789/0 conn(0x7f10bc0ae250 legacy=0x7f10bc13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:45.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:45.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 1 monclient: found mon.noname-b 2024-03-29T10:54:45.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:45.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:54:45.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc0aab80 con 0x7f10bc142800 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f10b40019f0 con 0x7f10bc142800 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.462+0000 7f10b3fff640 10 monclient: dump: 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:45.445 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.463+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b4001c50 con 0x7f10bc142800 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.463+0000 7f10b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.463+0000 7f10b3fff640 1 -- 172.21.15.155:0/158960789 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f10b40020a0 con 0x7f10bc142800 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.463+0000 7f10b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.463+0000 7f10b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.463+0000 7f10b3fff640 10 monclient: dump: 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.446 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:45.447 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:45.447 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:45.447 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:45.447 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:45.447 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.464+0000 7f10c379d640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.464+0000 7f10c379d640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.464+0000 7f10c379d640 4 set_mon_vals no callback set 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.465+0000 7f10c379d640 10 monclient: shutdown 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.465+0000 7f10c379d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.465+0000 7f10c379d640 1 -- 172.21.15.155:0/158960789 >> v1:172.21.15.155:6790/0 conn(0x7f10bc142800 legacy=0x7f10bc14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.465+0000 7f10c379d640 1 -- 172.21.15.155:0/158960789 shutdown_connections 2024-03-29T10:54:45.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.465+0000 7f10c379d640 1 -- 172.21.15.155:0/158960789 >> 172.21.15.155:0/158960789 conn(0x7f10bc0b35e0 msgr2=0x7f10bc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:45.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.469+0000 7f10c379d640 1 -- 172.21.15.155:0/158960789 shutdown_connections 2024-03-29T10:54:45.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.470+0000 7f10c379d640 1 -- 172.21.15.155:0/158960789 wait complete. 2024-03-29T10:54:45.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 10 monclient: build_initial_monmap 2024-03-29T10:54:45.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 10 monclient: monmap: 2024-03-29T10:54:45.453 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:45.453 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:45.453 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:45.453 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:45.453 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 1 librados: starting msgr at 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 1 librados: starting objecter 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 1 Processor -- start 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 1 -- start start 2024-03-29T10:54:45.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 1 librados: setting wanted keys 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 1 librados: calling monclient init 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.471+0000 7f10c379d640 10 monclient: init 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 10 monclient(hunting): picked mon.noname-c con 0x7f10bc0ae250 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 10 monclient(hunting): picked mon.noname-a con 0x7f10bc13f970 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 10 monclient(hunting): picked mon.noname-b con 0x7f10bc142800 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc1de910 con 0x7f10bc13f970 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc1463f0 con 0x7f10bc142800 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc1df4b0 con 0x7f10bc0ae250 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:45.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.472+0000 7f10c379d640 10 monclient(hunting): authenticate will time out at 988.765382s 2024-03-29T10:54:45.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.477+0000 7f10c1d3a640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f10bc142800 0x7f10bc071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:50533/0 (socket says 172.21.15.155:50533) 2024-03-29T10:54:45.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.477+0000 7f10c1d3a640 1 -- 172.21.15.155:0/1406794549 learned_addr learned my addr 172.21.15.155:0/1406794549 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.480+0000 7f10b1ffb640 10 client.?.objecter ms_handle_connect 0x7f10bc142800 2024-03-29T10:54:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.480+0000 7f10b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.480+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856533741 0 0) 0x7f10bc1463f0 con 0x7f10bc142800 2024-03-29T10:54:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.481+0000 7f10b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.481+0000 7f10b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:45.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.481+0000 7f10b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:45.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.481+0000 7f10b1ffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:54:45.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.481+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a0004eb0 con 0x7f10bc142800 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.481+0000 7f10b1ffb640 10 client.?.objecter ms_handle_connect 0x7f10bc0ae250 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.481+0000 7f10b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 client.?.objecter ms_handle_connect 0x7f10bc13f970 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3543879288 0 0) 0x7f10a0004eb0 con 0x7f10bc142800 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10bc1463f0 con 0x7f10bc142800 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619279847 0 0) 0x7f10bc1df4b0 con 0x7f10bc0ae250 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:45.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a0004eb0 con 0x7f10bc0ae250 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f10b40020a0 con 0x7f10bc142800 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:45.467 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820156801 0 0) 0x7f10bc1de910 con 0x7f10bc13f970 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10bc1df4b0 con 0x7f10bc13f970 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2571172602 0 0) 0x7f10a0004eb0 con 0x7f10bc0ae250 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10bc1de910 con 0x7f10bc0ae250 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.482+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f10b80018b0 con 0x7f10bc0ae250 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:45.468 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 878198737 0 0) 0x7f10bc1de910 con 0x7f10bc0ae250 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 >> v1:172.21.15.183:6789/0 conn(0x7f10bc142800 legacy=0x7f10bc071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 >> v1:172.21.15.155:6789/0 conn(0x7f10bc13f970 legacy=0x7f10bc1deda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:45.469 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:45.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 1 monclient: found mon.c 2024-03-29T10:54:45.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:45.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:45.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc1df640 con 0x7f10bc0ae250 2024-03-29T10:54:45.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.483+0000 7f10b1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.485+0000 7f10b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.485+0000 7f10b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.485+0000 7f10c379d640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.485+0000 7f10c379d640 10 monclient: _renew_subs 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.485+0000 7f10c379d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.485+0000 7f10c379d640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc13a890 con 0x7f10bc0ae250 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b8001cc0 con 0x7f10bc0ae250 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f10b80037b0 con 0x7f10bc0ae250 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10c379d640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:45.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10b1ffb640 10 monclient: dump: 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10c379d640 10 monclient: _renew_subs 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10c379d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10c379d640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc0724f0 con 0x7f10bc0ae250 2024-03-29T10:54:45.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.486+0000 7f10c379d640 1 librados: init done 2024-03-29T10:54:45.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.487+0000 7f10b37fe640 4 set_mon_vals no callback set 2024-03-29T10:54:45.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.487+0000 7f10a77fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:45.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.488+0000 7f10a77fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:45.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.489+0000 7f10a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:45.473 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.489+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1094005080 con 0x7f10bc0ae250 2024-03-29T10:54:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.490+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f10b8019780 con 0x7f10bc0ae250 2024-03-29T10:54:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.491+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f10b8083720 con 0x7f10bc0ae250 2024-03-29T10:54:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.491+0000 7f10b1ffb640 10 client.4343.objecter ms_dispatch 0x7f10bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.491+0000 7f10b1ffb640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.491+0000 7f10b1ffb640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:45.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.491+0000 7f10b1ffb640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T10:54:45.476 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.495+0000 7f10c0d38640 10 monclient: get_auth_request con 0x7f10a006dd50 auth_method 0 2024-03-29T10:54:45.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.497+0000 7f10b1ffb640 10 client.4343.objecter ms_handle_connect 0x7f10a006dd50 2024-03-29T10:54:45.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.498+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10b804f000 con 0x7f10bc0ae250 2024-03-29T10:54:45.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.498+0000 7f10b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:45.480 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.498+0000 7f10b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:45.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.828+0000 7f10a77fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:45.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.828+0000 7f10a77fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:45.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.828+0000 7f10a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:54:45.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.828+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1094005330 con 0x7f10bc0ae250 2024-03-29T10:54:45.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.831+0000 7f10b1ffb640 1 -- 172.21.15.155:0/1406794549 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f10b8050f30 con 0x7f10bc0ae250 2024-03-29T10:54:45.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.831+0000 7f10b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:45.813 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.831+0000 7f10b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:45.816 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:45.816 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:45.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.836+0000 7f10a77fe640 10 librados: watch_flush enter 2024-03-29T10:54:45.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.836+0000 7f10a77fe640 10 librados: watch_flush exit 2024-03-29T10:54:45.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.836+0000 7f10a77fe640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T10:54:45.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.836+0000 7f10a77fe640 10 client.4343.objecter successfully canceled tick 2024-03-29T10:54:45.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.837+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 >> v1:172.21.15.183:6816/1412921548 conn(0x7f10a006dd50 legacy=0x7f10a0070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:45.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.837+0000 7f10a77fe640 10 monclient: shutdown 2024-03-29T10:54:45.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.837+0000 7f10a77fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:45.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.837+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 >> v1:172.21.15.155:6790/0 conn(0x7f10bc0ae250 legacy=0x7f10bc144bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:45.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.838+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 shutdown_connections 2024-03-29T10:54:45.819 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.838+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 >> 172.21.15.155:0/1406794549 conn(0x7f10bc0b35e0 msgr2=0x7f10bc13e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:45.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.839+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 shutdown_connections 2024-03-29T10:54:45.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.841+0000 7f10a77fe640 1 -- 172.21.15.155:0/1406794549 wait complete. 2024-03-29T10:54:45.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:45.841+0000 7f10a77fe640 1 librados: shutdown 2024-03-29T10:54:45.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:52.647 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:52.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.819+0000 7f0fc1ff4640 10 monclient: get_monmap_and_config 2024-03-29T10:54:52.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.819+0000 7f0fc1ff4640 10 monclient: build_initial_monmap 2024-03-29T10:54:52.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.819+0000 7f0fc1ff4640 10 monclient: monmap: 2024-03-29T10:54:52.801 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:54:52.821547+0000 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:54:52.821547+0000 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:54:52.802 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:52.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.823+0000 7f0fc1ff4640 1 Processor -- start 2024-03-29T10:54:52.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.823+0000 7f0fc1ff4640 1 -- start start 2024-03-29T10:54:52.806 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.823+0000 7f0fc1ff4640 10 monclient: init 2024-03-29T10:54:52.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:52.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:52.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f0fbc1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:52.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f0fbc13f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:52.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f0fbc0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:52.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc0aaef0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc145bd0 con 0x7f0fbc1457f0 2024-03-29T10:54:52.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc145eb0 con 0x7f0fbc13f650 2024-03-29T10:54:52.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:52.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.824+0000 7f0fc1ff4640 10 monclient(hunting): authenticate will time out at 996.117646s 2024-03-29T10:54:52.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.829+0000 7f0fbbfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f0fbc0a9d70 0x7f0fbc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33357/0 (socket says 172.21.15.155:33357) 2024-03-29T10:54:52.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.829+0000 7f0fbbfff640 1 -- 172.21.15.155:0/1546814312 learned_addr learned my addr 172.21.15.155:0/1546814312 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:52.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095670653 0 0) 0x7f0fbc0aaef0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:52.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:52.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:52.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:54:52.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa4003520 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.814 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434138887 0 0) 0x7f0fbc145bd0 con 0x7f0fbc1457f0 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa40048a0 con 0x7f0fbc1457f0 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698311566 0 0) 0x7f0fbc145eb0 con 0x7f0fbc13f650 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:54:52.815 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.832+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fbc145bd0 con 0x7f0fbc13f650 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490352807 0 0) 0x7f0fbc145bd0 con 0x7f0fbc13f650 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 >> v1:172.21.15.183:6789/0 conn(0x7f0fbc1457f0 legacy=0x7f0fbc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 >> v1:172.21.15.155:6789/0 conn(0x7f0fbc0a9d70 legacy=0x7f0fbc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 monclient: found mon.noname-b 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc146340 con 0x7f0fbc13f650 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:52.816 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0fb0001550 con 0x7f0fbc13f650 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: dump: 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:52.817 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fb00016b0 con 0x7f0fbc13f650 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 1 -- 172.21.15.155:0/1546814312 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0fb0001ac0 con 0x7f0fbc13f650 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.833+0000 7f0fba7fc640 10 monclient: dump: 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.818 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.819 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:52.819 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:52.819 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:52.819 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:52.819 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:52.819 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.834+0000 7f0fc1ff4640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.834+0000 7f0fc1ff4640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.834+0000 7f0fc1ff4640 4 set_mon_vals no callback set 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.835+0000 7f0fc1ff4640 10 monclient: shutdown 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.835+0000 7f0fc1ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.835+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/1546814312 >> v1:172.21.15.155:6790/0 conn(0x7f0fbc13f650 legacy=0x7f0fbc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.836+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/1546814312 shutdown_connections 2024-03-29T10:54:52.820 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.836+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/1546814312 >> 172.21.15.155:0/1546814312 conn(0x7f0fbc066a00 msgr2=0x7f0fbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:52.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.836+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/1546814312 shutdown_connections 2024-03-29T10:54:52.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.836+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/1546814312 wait complete. 2024-03-29T10:54:52.821 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.837+0000 7f0fc1ff4640 10 monclient: build_initial_monmap 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.837+0000 7f0fc1ff4640 10 monclient: monmap: 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.838+0000 7f0fc1ff4640 1 librados: starting msgr at 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.838+0000 7f0fc1ff4640 1 librados: starting objecter 2024-03-29T10:54:52.822 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.838+0000 7f0fc1ff4640 1 Processor -- start 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.838+0000 7f0fc1ff4640 1 -- start start 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.838+0000 7f0fc1ff4640 1 librados: setting wanted keys 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.838+0000 7f0fc1ff4640 1 librados: calling monclient init 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.838+0000 7f0fc1ff4640 10 monclient: init 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 10 monclient(hunting): picked mon.noname-a con 0x7f0fbc0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 10 monclient(hunting): picked mon.noname-c con 0x7f0fbc13f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f0fbc1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc06e710 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc06e980 con 0x7f0fbc1457f0 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fbc131bc0 con 0x7f0fbc13f650 2024-03-29T10:54:52.823 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:52.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.839+0000 7f0fc1ff4640 10 monclient(hunting): authenticate will time out at 996.132296s 2024-03-29T10:54:52.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.846+0000 7f0fbb7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f0fbc0a9d70 0x7f0fbc1320e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33429/0 (socket says 172.21.15.155:33429) 2024-03-29T10:54:52.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.846+0000 7f0fbb7fe640 1 -- 172.21.15.155:0/131032413 learned_addr learned my addr 172.21.15.155:0/131032413 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 10 client.?.objecter ms_handle_connect 0x7f0fbc0a9d70 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140968431 0 0) 0x7f0fbc06e710 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:54:52.830 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.848+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa8004eb0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 client.?.objecter ms_handle_connect 0x7f0fbc1457f0 2024-03-29T10:54:52.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:52.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 client.?.objecter ms_handle_connect 0x7f0fbc13f650 2024-03-29T10:54:52.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:52.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874802842 0 0) 0x7f0fa8004eb0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fbc06e710 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0fb4001cf0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): dump: 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.841 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347792747 0 0) 0x7f0fbc131bc0 con 0x7f0fbc13f650 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa8004eb0 con 0x7f0fbc13f650 2024-03-29T10:54:52.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3808711841 0 0) 0x7f0fbc06e710 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 >> v1:172.21.15.155:6790/0 conn(0x7f0fbc13f650 legacy=0x7f0fbc08ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 >> v1:172.21.15.183:6789/0 conn(0x7f0fbc1457f0 legacy=0x7f0fbc1ef360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 monclient: found mon.a 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc13fda0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.849+0000 7f0fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 10 monclient: _renew_subs 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fbc1327f0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 10 monclient: _renew_subs 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fbc1447c0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fc1ff4640 1 librados: init done 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fb4003990 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0fb4003de0 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fa3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fa3fff640 10 monclient: dump: 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.844 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:54:52.845 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:52.845 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:54:52.845 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:54:52.845 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:54:52.845 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:54:52.845 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:54:52.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.850+0000 7f0fb9ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:52.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.852+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f0fb4003f40 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.853+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f0fb4084e50 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.853+0000 7f0fa3fff640 10 client.4326.objecter ms_dispatch 0x7f0fbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:52.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.853+0000 7f0fa3fff640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:52.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.853+0000 7f0fa3fff640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:52.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.853+0000 7f0fa3fff640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T10:54:52.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.853+0000 7f0fbaffd640 10 monclient: get_auth_request con 0x7f0fa806dd60 auth_method 0 2024-03-29T10:54:52.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.854+0000 7f0fa3fff640 10 client.4326.objecter ms_handle_connect 0x7f0fa806dd60 2024-03-29T10:54:52.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.855+0000 7f0fc1ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:52.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.855+0000 7f0fc1ff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:52.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.855+0000 7f0fc1ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:52.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.855+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c005080 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.865+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fb4050730 con 0x7f0fbc0a9d70 2024-03-29T10:54:52.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.865+0000 7f0fa3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:52.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:52.865+0000 7f0fa3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:53.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.189+0000 7f0fc1ff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:53.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.189+0000 7f0fc1ff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:53.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.189+0000 7f0fc1ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:54:53.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.189+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f9c005330 con 0x7f0fbc0a9d70 2024-03-29T10:54:53.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.190+0000 7f0fa3fff640 1 -- 172.21.15.155:0/131032413 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f0fb4052660 con 0x7f0fbc0a9d70 2024-03-29T10:54:53.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.190+0000 7f0fa3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:53.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.190+0000 7f0fa3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:53.175 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:54:53.176 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:53.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.194+0000 7f0fc1ff4640 10 librados: watch_flush enter 2024-03-29T10:54:53.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.194+0000 7f0fc1ff4640 10 librados: watch_flush exit 2024-03-29T10:54:53.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.194+0000 7f0fc1ff4640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T10:54:53.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.194+0000 7f0fc1ff4640 10 client.4326.objecter successfully canceled tick 2024-03-29T10:54:53.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.195+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 >> v1:172.21.15.183:6816/1412921548 conn(0x7f0fa806dd60 legacy=0x7f0fa8070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:53.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.195+0000 7f0fc1ff4640 10 monclient: shutdown 2024-03-29T10:54:53.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.195+0000 7f0fc1ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:53.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.195+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 >> v1:172.21.15.155:6789/0 conn(0x7f0fbc0a9d70 legacy=0x7f0fbc1320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:53.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.195+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 shutdown_connections 2024-03-29T10:54:53.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.195+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 >> 172.21.15.155:0/131032413 conn(0x7f0fbc066a00 msgr2=0x7f0fbc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:53.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.198+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 shutdown_connections 2024-03-29T10:54:53.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.198+0000 7f0fc1ff4640 1 -- 172.21.15.155:0/131032413 wait complete. 2024-03-29T10:54:53.180 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:54:53.199+0000 7f0fc1ff4640 1 librados: shutdown 2024-03-29T10:54:53.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:00.008 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:00.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.179+0000 7fcbd569a640 10 monclient: get_monmap_and_config 2024-03-29T10:55:00.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.179+0000 7fcbd569a640 10 monclient: build_initial_monmap 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.179+0000 7fcbd569a640 10 monclient: monmap: 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:00.180512+0000 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:00.180512+0000 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:00.162 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:00.163 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.180+0000 7fcbd569a640 1 Processor -- start 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.180+0000 7fcbd569a640 1 -- start start 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.180+0000 7fcbd569a640 10 monclient: init 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 10 monclient(hunting): picked mon.noname-c con 0x7fcbd006ab70 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 10 monclient(hunting): picked mon.noname-b con 0x7fcbd006af50 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:00.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 10 monclient(hunting): picked mon.noname-a con 0x7fcbd00b5320 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:00.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd00b4e90 con 0x7fcbd00b5320 2024-03-29T10:55:00.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd00703b0 con 0x7fcbd006ab70 2024-03-29T10:55:00.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd013f830 con 0x7fcbd006af50 2024-03-29T10:55:00.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:00.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.181+0000 7fcbd569a640 10 monclient(hunting): authenticate will time out at 1003.473621s 2024-03-29T10:55:00.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.185+0000 7fcbce7fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fcbd006af50 0x7fcbd013f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:46459/0 (socket says 172.21.15.155:46459) 2024-03-29T10:55:00.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.185+0000 7fcbce7fc640 1 -- 172.21.15.155:0/373792987 learned_addr learned my addr 172.21.15.155:0/373792987 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:00.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.186+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404528681 0 0) 0x7fcbd013f830 con 0x7fcbd006af50 2024-03-29T10:55:00.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.186+0000 7fcbcdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.186+0000 7fcbcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.186+0000 7fcbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.186+0000 7fcbcdffb640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.186+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb8003520 con 0x7fcbd006af50 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.187+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 485493509 0 0) 0x7fcbb8003520 con 0x7fcbd006af50 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 >> v1:172.21.15.183:6789/0 conn(0x7fcbd006ab70 legacy=0x7fcbd0141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 >> v1:172.21.15.155:6789/0 conn(0x7fcbd00b5320 legacy=0x7fcbd014d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:00.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 1 monclient: found mon.noname-b 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd00b5770 con 0x7fcbd006af50 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbd569a640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:55:00.173 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbd569a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcbc8001a10 con 0x7fcbd006af50 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.188+0000 7fcbcdffb640 10 monclient: dump: 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.174 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.189+0000 7fcbd569a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.189+0000 7fcbcdffb640 1 -- 172.21.15.155:0/373792987 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbc8001ca0 con 0x7fcbd006af50 2024-03-29T10:55:00.175 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.189+0000 7fcbcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:00.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.189+0000 7fcbd569a640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:00.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.189+0000 7fcbd569a640 4 set_mon_vals no callback set 2024-03-29T10:55:00.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.190+0000 7fcbd569a640 10 monclient: shutdown 2024-03-29T10:55:00.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.190+0000 7fcbd569a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:00.176 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.190+0000 7fcbd569a640 1 -- 172.21.15.155:0/373792987 >> v1:172.21.15.155:6790/0 conn(0x7fcbd006af50 legacy=0x7fcbd013f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:00.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.192+0000 7fcbd569a640 1 -- 172.21.15.155:0/373792987 shutdown_connections 2024-03-29T10:55:00.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.192+0000 7fcbd569a640 1 -- 172.21.15.155:0/373792987 >> 172.21.15.155:0/373792987 conn(0x7fcbd00b9540 msgr2=0x7fcbd00b9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:00.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.193+0000 7fcbd569a640 1 -- 172.21.15.155:0/373792987 shutdown_connections 2024-03-29T10:55:00.177 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.197+0000 7fcbd569a640 1 -- 172.21.15.155:0/373792987 wait complete. 2024-03-29T10:55:00.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.198+0000 7fcbd569a640 10 monclient: build_initial_monmap 2024-03-29T10:55:00.178 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.198+0000 7fcbd569a640 10 monclient: monmap: 2024-03-29T10:55:00.178 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:00.179 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:00.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.198+0000 7fcbd569a640 1 librados: starting msgr at 2024-03-29T10:55:00.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.198+0000 7fcbd569a640 1 librados: starting objecter 2024-03-29T10:55:00.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.199+0000 7fcbd569a640 1 Processor -- start 2024-03-29T10:55:00.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.200+0000 7fcbd569a640 1 -- start start 2024-03-29T10:55:00.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.200+0000 7fcbd569a640 1 librados: setting wanted keys 2024-03-29T10:55:00.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.200+0000 7fcbd569a640 1 librados: calling monclient init 2024-03-29T10:55:00.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.200+0000 7fcbd569a640 10 monclient: init 2024-03-29T10:55:00.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:00.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:00.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 10 monclient(hunting): picked mon.noname-b con 0x7fcbd00b5320 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:00.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 10 monclient(hunting): picked mon.noname-c con 0x7fcbd01db720 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:00.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 10 monclient(hunting): picked mon.noname-a con 0x7fcbd013ca40 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:00.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd01dc290 con 0x7fcbd013ca40 2024-03-29T10:55:00.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd0146010 con 0x7fcbd00b5320 2024-03-29T10:55:00.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd0137f90 con 0x7fcbd01db720 2024-03-29T10:55:00.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:00.185 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.201+0000 7fcbd569a640 10 monclient(hunting): authenticate will time out at 1003.493103s 2024-03-29T10:55:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.204+0000 7fcbceffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fcbd00b5320 0x7fcbd0144750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:53815/0 (socket says 172.21.15.155:53815) 2024-03-29T10:55:00.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.204+0000 7fcbceffd640 1 -- 172.21.15.155:0/1713486562 learned_addr learned my addr 172.21.15.155:0/1713486562 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:00.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.207+0000 7fcbb77fe640 10 client.?.objecter ms_handle_connect 0x7fcbd00b5320 2024-03-29T10:55:00.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.208+0000 7fcbb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:00.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.208+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371199500 0 0) 0x7fcbd0146010 con 0x7fcbd00b5320 2024-03-29T10:55:00.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:00.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:00.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:00.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:55:00.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbbc004eb0 con 0x7fcbd00b5320 2024-03-29T10:55:00.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 client.?.objecter ms_handle_connect 0x7fcbd013ca40 2024-03-29T10:55:00.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:00.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001542653 0 0) 0x7fcbd01dc290 con 0x7fcbd013ca40 2024-03-29T10:55:00.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:00.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:00.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:00.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:55:00.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.209+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd0146010 con 0x7fcbd013ca40 2024-03-29T10:55:00.199 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2356452722 0 0) 0x7fcbbc004eb0 con 0x7fcbd00b5320 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbd01dc290 con 0x7fcbd00b5320 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcbc00019d0 con 0x7fcbd00b5320 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 10 monclient(hunting): dump: 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:00.200 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2537142457 0 0) 0x7fcbd0146010 con 0x7fcbd013ca40 2024-03-29T10:55:00.201 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbbc004eb0 con 0x7fcbd013ca40 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.210+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcbc4001780 con 0x7fcbd013ca40 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 10 monclient(hunting): dump: 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.202 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1418864899 0 0) 0x7fcbd01dc290 con 0x7fcbd00b5320 2024-03-29T10:55:00.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 >> v1:172.21.15.155:6790/0 conn(0x7fcbd01db720 legacy=0x7fcbd01dbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 >> v1:172.21.15.155:6789/0 conn(0x7fcbd013ca40 legacy=0x7fcbd013ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 1 monclient: found mon.b 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd0144e60 con 0x7fcbd00b5320 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:00.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.211+0000 7fcbd569a640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 10 monclient: _renew_subs 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd013e800 con 0x7fcbd00b5320 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 10 monclient: _renew_subs 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd013d560 con 0x7fcbd00b5320 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbd569a640 1 librados: init done 2024-03-29T10:55:00.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbc0003610 con 0x7fcbd00b5320 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcbc0003a20 con 0x7fcbd00b5320 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbb77fe640 10 monclient: dump: 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:00.206 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.212+0000 7fcbcd7fa640 4 set_mon_vals no callback set 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.214+0000 7fcbd569a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.215+0000 7fcbd569a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.215+0000 7fcbd569a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.215+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb0005080 con 0x7fcbd00b5320 2024-03-29T10:55:00.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.217+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fcbc0003b80 con 0x7fcbd00b5320 2024-03-29T10:55:00.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.220+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fcbc0084aa0 con 0x7fcbd00b5320 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.220+0000 7fcbb77fe640 10 client.4336.objecter ms_dispatch 0x7fcbd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.220+0000 7fcbb77fe640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.220+0000 7fcbce7fc640 10 monclient: get_auth_request con 0x7fcbbc06de00 auth_method 0 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.220+0000 7fcbb77fe640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.220+0000 7fcbb77fe640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.220+0000 7fcbb77fe640 10 client.4336.objecter ms_handle_connect 0x7fcbbc06de00 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.224+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbc0050380 con 0x7fcbd00b5320 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.224+0000 7fcbb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:00.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.224+0000 7fcbb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:00.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.546+0000 7fcbd569a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:00.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.546+0000 7fcbd569a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:00.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.546+0000 7fcbd569a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:00.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.546+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbb0005330 con 0x7fcbd00b5320 2024-03-29T10:55:00.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.549+0000 7fcbb77fe640 1 -- 172.21.15.155:0/1713486562 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fcbc00522b0 con 0x7fcbd00b5320 2024-03-29T10:55:00.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.549+0000 7fcbb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:00.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.549+0000 7fcbb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:00.532 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:00.532 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:00.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 10 librados: watch_flush enter 2024-03-29T10:55:00.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 10 librados: watch_flush exit 2024-03-29T10:55:00.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T10:55:00.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 10 client.4336.objecter successfully canceled tick 2024-03-29T10:55:00.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 >> v1:172.21.15.183:6816/1412921548 conn(0x7fcbbc06de00 legacy=0x7fcbbc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:00.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 10 monclient: shutdown 2024-03-29T10:55:00.536 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:00.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.555+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 >> v1:172.21.15.183:6789/0 conn(0x7fcbd00b5320 legacy=0x7fcbd0144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:00.537 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.557+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 shutdown_connections 2024-03-29T10:55:00.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.557+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 >> 172.21.15.155:0/1713486562 conn(0x7fcbd00b9540 msgr2=0x7fcbd00b87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:00.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.557+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 shutdown_connections 2024-03-29T10:55:00.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.557+0000 7fcbd569a640 1 -- 172.21.15.155:0/1713486562 wait complete. 2024-03-29T10:55:00.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:00.558+0000 7fcbd569a640 1 librados: shutdown 2024-03-29T10:55:00.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:07.366 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:07.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.521+0000 7f5a3b9ef640 10 monclient: get_monmap_and_config 2024-03-29T10:55:07.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.521+0000 7f5a3b9ef640 10 monclient: build_initial_monmap 2024-03-29T10:55:07.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.521+0000 7f5a3b9ef640 10 monclient: monmap: 2024-03-29T10:55:07.502 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:07.522350+0000 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:07.522350+0000 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:07.503 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:07.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.522+0000 7f5a3b9ef640 1 Processor -- start 2024-03-29T10:55:07.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.523+0000 7f5a3b9ef640 1 -- start start 2024-03-29T10:55:07.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.523+0000 7f5a3b9ef640 10 monclient: init 2024-03-29T10:55:07.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 10 monclient(hunting): picked mon.noname-c con 0x7f5a341457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 10 monclient(hunting): picked mon.noname-a con 0x7f5a3413f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5a340a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a340aaef0 con 0x7f5a3413f650 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a34145bd0 con 0x7f5a341457f0 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a34145eb0 con 0x7f5a340a9d70 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a3b9ef640 10 monclient(hunting): authenticate will time out at 1010.816532s 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a39f8c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5a340a9d70 0x7f5a3414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:54032/0 (socket says 172.21.15.155:54032) 2024-03-29T10:55:07.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.524+0000 7f5a39f8c640 1 -- 172.21.15.155:0/1520433125 learned_addr learned my addr 172.21.15.155:0/1520433125 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502045494 0 0) 0x7f5a34145eb0 con 0x7f5a340a9d70 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a20003520 con 0x7f5a340a9d70 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694645094 0 0) 0x7f5a20003520 con 0x7f5a340a9d70 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.525+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 >> v1:172.21.15.183:6789/0 conn(0x7f5a341457f0 legacy=0x7f5a3413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 >> v1:172.21.15.155:6789/0 conn(0x7f5a3413f650 legacy=0x7f5a340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 1 monclient: found mon.noname-b 2024-03-29T10:55:07.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a34146340 con 0x7f5a340a9d70 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 10 monclient: _finish_auth 0 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a3b9ef640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5a24001690 con 0x7f5a340a9d70 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a3b9ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a2bfff640 10 monclient: dump: 2024-03-29T10:55:07.513 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.526+0000 7f5a3b9ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a24001860 con 0x7f5a340a9d70 2024-03-29T10:55:07.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a2bfff640 1 -- 172.21.15.155:0/1520433125 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5a24001c70 con 0x7f5a340a9d70 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a2bfff640 10 monclient: dump: 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:07.515 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a3b9ef640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.527+0000 7f5a3b9ef640 4 set_mon_vals no callback set 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.528+0000 7f5a3b9ef640 10 monclient: shutdown 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.528+0000 7f5a3b9ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.528+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/1520433125 >> v1:172.21.15.155:6790/0 conn(0x7f5a340a9d70 legacy=0x7f5a3414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.528+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/1520433125 shutdown_connections 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.528+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/1520433125 >> 172.21.15.155:0/1520433125 conn(0x7f5a34066a00 msgr2=0x7f5a34066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.529+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/1520433125 shutdown_connections 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.529+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/1520433125 wait complete. 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 10 monclient: build_initial_monmap 2024-03-29T10:55:07.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 10 monclient: monmap: 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 1 librados: starting msgr at 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 1 librados: starting objecter 2024-03-29T10:55:07.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 1 Processor -- start 2024-03-29T10:55:07.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 1 -- start start 2024-03-29T10:55:07.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 1 librados: setting wanted keys 2024-03-29T10:55:07.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 1 librados: calling monclient init 2024-03-29T10:55:07.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.530+0000 7f5a3b9ef640 10 monclient: init 2024-03-29T10:55:07.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:07.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:07.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 10 monclient(hunting): picked mon.noname-a con 0x7f5a340a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:07.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5a3413f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:07.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 10 monclient(hunting): picked mon.noname-c con 0x7f5a341457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:07.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a341421a0 con 0x7f5a340a9d70 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a34137a20 con 0x7f5a3413f650 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a340abf30 con 0x7f5a341457f0 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3b9ef640 10 monclient(hunting): authenticate will time out at 1010.823456s 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3978b640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5a340a9d70 0x7f5a34142690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:32772/0 (socket says 172.21.15.155:32772) 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.531+0000 7f5a3978b640 1 -- 172.21.15.155:0/2152613932 learned_addr learned my addr 172.21.15.155:0/2152613932 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 client.?.objecter ms_handle_connect 0x7f5a340a9d70 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 client.?.objecter ms_handle_connect 0x7f5a341457f0 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243626338 0 0) 0x7f5a340abf30 con 0x7f5a341457f0 2024-03-29T10:55:07.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a18004eb0 con 0x7f5a341457f0 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303243714 0 0) 0x7f5a341421a0 con 0x7f5a340a9d70 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a340abf30 con 0x7f5a340a9d70 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1315931827 0 0) 0x7f5a18004eb0 con 0x7f5a341457f0 2024-03-29T10:55:07.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a341421a0 con 0x7f5a341457f0 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5a24001b70 con 0x7f5a341457f0 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:07.525 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.532+0000 7f5a29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 225591584 0 0) 0x7f5a340abf30 con 0x7f5a340a9d70 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a18004eb0 con 0x7f5a340a9d70 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5a2c001790 con 0x7f5a340a9d70 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:07.526 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3191312420 0 0) 0x7f5a341421a0 con 0x7f5a341457f0 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.533+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 >> v1:172.21.15.183:6789/0 conn(0x7f5a3413f650 legacy=0x7f5a34136c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 >> v1:172.21.15.155:6789/0 conn(0x7f5a340a9d70 legacy=0x7f5a34142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:07.527 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 1 monclient: found mon.c 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a340ac210 con 0x7f5a341457f0 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 5 monclient: authenticate success, global_id 4358 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 10 monclient: _renew_subs 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a340ac4f0 con 0x7f5a341457f0 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:07.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 10 monclient: _renew_subs 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3413fda0 con 0x7f5a341457f0 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.535+0000 7f5a3b9ef640 1 librados: init done 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.536+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a24001010 con 0x7f5a341457f0 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.536+0000 7f5a29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.536+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5a24004bf0 con 0x7f5a341457f0 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.536+0000 7f5a29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.536+0000 7f5a29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.536+0000 7f5a29ffb640 10 monclient: dump: 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:07.529 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.536+0000 7f5a2b7fe640 4 set_mon_vals no callback set 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.537+0000 7f5a3b9ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.538+0000 7f5a3b9ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.538+0000 7f5a3b9ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:07.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.538+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a14005080 con 0x7f5a341457f0 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.538+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f5a2401abc0 con 0x7f5a341457f0 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.539+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f5a24084aa0 con 0x7f5a341457f0 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.539+0000 7f5a29ffb640 10 client.4358.objecter ms_dispatch 0x7f5a34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.539+0000 7f5a29ffb640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.539+0000 7f5a29ffb640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.539+0000 7f5a29ffb640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.539+0000 7f5a3978b640 10 monclient: get_auth_request con 0x7f5a1806dd60 auth_method 0 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.540+0000 7f5a29ffb640 10 client.4358.objecter ms_handle_connect 0x7f5a1806dd60 2024-03-29T10:55:07.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.547+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a24050380 con 0x7f5a341457f0 2024-03-29T10:55:07.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.547+0000 7f5a29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:07.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.547+0000 7f5a29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:07.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.855+0000 7f5a3b9ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:07.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.855+0000 7f5a3b9ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:07.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.855+0000 7f5a3b9ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:07.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.855+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a14005330 con 0x7f5a341457f0 2024-03-29T10:55:07.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.856+0000 7f5a29ffb640 1 -- 172.21.15.155:0/2152613932 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f5a240522b0 con 0x7f5a341457f0 2024-03-29T10:55:07.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.856+0000 7f5a29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:07.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.856+0000 7f5a29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:07.840 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:07.840 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:07.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.860+0000 7f5a3b9ef640 10 librados: watch_flush enter 2024-03-29T10:55:07.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.860+0000 7f5a3b9ef640 10 librados: watch_flush exit 2024-03-29T10:55:07.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.860+0000 7f5a3b9ef640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T10:55:07.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.860+0000 7f5a3b9ef640 10 client.4358.objecter successfully canceled tick 2024-03-29T10:55:07.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.861+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 >> v1:172.21.15.183:6816/1412921548 conn(0x7f5a1806dd60 legacy=0x7f5a18070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:07.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.861+0000 7f5a3b9ef640 10 monclient: shutdown 2024-03-29T10:55:07.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.861+0000 7f5a3b9ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:07.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.861+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 >> v1:172.21.15.155:6790/0 conn(0x7f5a341457f0 legacy=0x7f5a34137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:07.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.861+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 shutdown_connections 2024-03-29T10:55:07.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.861+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 >> 172.21.15.155:0/2152613932 conn(0x7f5a34066a00 msgr2=0x7f5a3413a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:07.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.864+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 shutdown_connections 2024-03-29T10:55:07.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.864+0000 7f5a3b9ef640 1 -- 172.21.15.155:0/2152613932 wait complete. 2024-03-29T10:55:07.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:07.865+0000 7f5a3b9ef640 1 librados: shutdown 2024-03-29T10:55:07.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:14.670 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:14.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.852+0000 7efd34372640 10 monclient: get_monmap_and_config 2024-03-29T10:55:14.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.852+0000 7efd34372640 10 monclient: build_initial_monmap 2024-03-29T10:55:14.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.852+0000 7efd34372640 10 monclient: monmap: 2024-03-29T10:55:14.833 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:14.853351+0000 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:14.853351+0000 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:14.834 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:14.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.853+0000 7efd34372640 1 Processor -- start 2024-03-29T10:55:14.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.853+0000 7efd34372640 1 -- start start 2024-03-29T10:55:14.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.854+0000 7efd34372640 10 monclient: init 2024-03-29T10:55:14.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.855+0000 7efd34372640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:14.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.855+0000 7efd34372640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:14.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.855+0000 7efd34372640 10 monclient(hunting): picked mon.noname-b con 0x7efd2c0b0860 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:14.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.855+0000 7efd34372640 10 monclient(hunting): picked mon.noname-a con 0x7efd2c13ef30 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:14.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.855+0000 7efd3210e640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7efd2c0b0860 0x7efd2c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53456/0 (socket says 172.21.15.155:53456) 2024-03-29T10:55:14.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3210e640 1 -- 172.21.15.155:0/2559261359 learned_addr learned my addr 172.21.15.155:0/2559261359 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:14.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd34372640 10 monclient(hunting): picked mon.noname-c con 0x7efd2c0af8e0 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:14.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c0b0c40 con 0x7efd2c13ef30 2024-03-29T10:55:14.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c0aed10 con 0x7efd2c0af8e0 2024-03-29T10:55:14.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c0afd50 con 0x7efd2c0b0860 2024-03-29T10:55:14.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd34372640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:14.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd34372640 10 monclient(hunting): authenticate will time out at 1018.148092s 2024-03-29T10:55:14.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207333954 0 0) 0x7efd2c0afd50 con 0x7efd2c0b0860 2024-03-29T10:55:14.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): _negotiate 2024-03-29T10:55:14.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:14.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:14.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd18003520 con 0x7efd2c0b0860 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497602585 0 0) 0x7efd2c0aed10 con 0x7efd2c0af8e0 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): _negotiate 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.856+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2c0afd50 con 0x7efd2c0af8e0 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060496278 0 0) 0x7efd2c0b0c40 con 0x7efd2c13ef30 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient(hunting): _negotiate 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:55:14.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd18004bb0 con 0x7efd2c13ef30 2024-03-29T10:55:14.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3012741876 0 0) 0x7efd18004bb0 con 0x7efd2c13ef30 2024-03-29T10:55:14.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 >> v1:172.21.15.155:6790/0 conn(0x7efd2c0b0860 legacy=0x7efd2c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:14.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 >> v1:172.21.15.183:6789/0 conn(0x7efd2c0af8e0 legacy=0x7efd2c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient: _finish_hunting 0 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 monclient: found mon.noname-a 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c0ab1b0 con 0x7efd2c13ef30 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient: _finish_auth 0 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient: _check_auth_tickets 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7efd1c0015b0 con 0x7efd2c13ef30 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:14.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd3110c640 10 monclient: dump: 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.857+0000 7efd34372640 5 monclient: authenticate success, global_id 4341 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.858+0000 7efd34372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.858+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd1c001790 con 0x7efd2c13ef30 2024-03-29T10:55:14.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.858+0000 7efd3110c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.858+0000 7efd3110c640 1 -- 172.21.15.155:0/2559261359 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7efd1c0015b0 con 0x7efd2c13ef30 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.858+0000 7efd3110c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.858+0000 7efd3110c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.858+0000 7efd3110c640 10 monclient: dump: 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:14.844 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.859+0000 7efd34372640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.859+0000 7efd34372640 4 set_mon_vals no callback set 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.860+0000 7efd34372640 10 monclient: shutdown 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.860+0000 7efd34372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.860+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 >> v1:172.21.15.155:6789/0 conn(0x7efd2c13ef30 legacy=0x7efd2c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.860+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 shutdown_connections 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.860+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 >> 172.21.15.155:0/2559261359 conn(0x7efd2c06b660 msgr2=0x7efd2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.860+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 shutdown_connections 2024-03-29T10:55:14.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.860+0000 7efd34372640 1 -- 172.21.15.155:0/2559261359 wait complete. 2024-03-29T10:55:14.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.861+0000 7efd34372640 10 monclient: build_initial_monmap 2024-03-29T10:55:14.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.861+0000 7efd34372640 10 monclient: monmap: 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.862+0000 7efd34372640 1 librados: starting msgr at 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.862+0000 7efd34372640 1 librados: starting objecter 2024-03-29T10:55:14.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.862+0000 7efd34372640 1 Processor -- start 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.862+0000 7efd34372640 1 -- start start 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.862+0000 7efd34372640 1 librados: setting wanted keys 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.862+0000 7efd34372640 1 librados: calling monclient init 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.862+0000 7efd34372640 10 monclient: init 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.863+0000 7efd34372640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.863+0000 7efd34372640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 10 monclient(hunting): picked mon.noname-c con 0x7efd2c0af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd3210e640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7efd2c0af8e0 0x7efd2c137e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53652/0 (socket says 172.21.15.155:53652) 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd3210e640 1 -- 172.21.15.155:0/413262574 learned_addr learned my addr 172.21.15.155:0/413262574 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 10 monclient(hunting): picked mon.noname-b con 0x7efd2c0b0860 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 10 monclient(hunting): picked mon.noname-a con 0x7efd2c13ef30 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c138510 con 0x7efd2c13ef30 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c1c8270 con 0x7efd2c0b0860 2024-03-29T10:55:14.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c1c8620 con 0x7efd2c0af8e0 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd34372640 10 monclient(hunting): authenticate will time out at 1018.156645s 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 client.?.objecter ms_handle_connect 0x7efd2c0af8e0 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 client.?.objecter ms_handle_connect 0x7efd2c0b0860 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348245801 0 0) 0x7efd2c1c8270 con 0x7efd2c0b0860 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.864+0000 7efd22ffd640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd10004eb0 con 0x7efd2c0b0860 2024-03-29T10:55:14.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 client.?.objecter ms_handle_connect 0x7efd2c13ef30 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696749565 0 0) 0x7efd2c1c8620 con 0x7efd2c0af8e0 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2c1c8270 con 0x7efd2c0af8e0 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987786400 0 0) 0x7efd10004eb0 con 0x7efd2c0b0860 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd2c1c8620 con 0x7efd2c0b0860 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7efd1c0015b0 con 0x7efd2c0b0860 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:14.856 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.865+0000 7efd22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:14.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267412461 0 0) 0x7efd2c0afd50 con 0x7efd2c13ef30 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd10004eb0 con 0x7efd2c13ef30 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4147064728 0 0) 0x7efd2c1c8270 con 0x7efd2c0af8e0 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd2c0afd50 con 0x7efd2c0af8e0 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7efd280018d0 con 0x7efd2c0af8e0 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:14.858 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2814241935 0 0) 0x7efd2c1c8620 con 0x7efd2c0b0860 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 >> v1:172.21.15.155:6790/0 conn(0x7efd2c0af8e0 legacy=0x7efd2c137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 >> v1:172.21.15.155:6789/0 conn(0x7efd2c13ef30 legacy=0x7efd2c1d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 monclient: found mon.b 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:14.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.866+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c1d0e40 con 0x7efd2c0b0860 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 5 monclient: authenticate success, global_id 4342 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 10 monclient: _renew_subs 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2c1c6b30 con 0x7efd2c0b0860 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 10 monclient: _renew_subs 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:14.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2c1c9560 con 0x7efd2c0b0860 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd34372640 1 librados: init done 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd1c001d60 con 0x7efd2c0b0860 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7efd1c003a40 con 0x7efd2c0b0860 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd23fff640 4 set_mon_vals no callback set 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.867+0000 7efd22ffd640 10 monclient: dump: 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.861 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.870+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7efd1c019a10 con 0x7efd2c0b0860 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.870+0000 7efd34372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.870+0000 7efd34372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.870+0000 7efd34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.870+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd14005080 con 0x7efd2c0b0860 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.871+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7efd1c083870 con 0x7efd2c0b0860 2024-03-29T10:55:14.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.871+0000 7efd22ffd640 10 client.4342.objecter ms_dispatch 0x7efd2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.871+0000 7efd22ffd640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.871+0000 7efd22ffd640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.871+0000 7efd22ffd640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T10:55:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.871+0000 7efd3210e640 10 monclient: get_auth_request con 0x7efd1006e000 auth_method 0 2024-03-29T10:55:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.871+0000 7efd22ffd640 10 client.4342.objecter ms_handle_connect 0x7efd1006e000 2024-03-29T10:55:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.877+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd1c04f150 con 0x7efd2c0b0860 2024-03-29T10:55:14.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.877+0000 7efd22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:14.864 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:14.877+0000 7efd22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:15.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.315+0000 7efd34372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:15.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.315+0000 7efd34372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:15.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.315+0000 7efd34372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:15.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.315+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd14005330 con 0x7efd2c0b0860 2024-03-29T10:55:15.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.316+0000 7efd22ffd640 1 -- 172.21.15.155:0/413262574 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7efd1c051080 con 0x7efd2c0b0860 2024-03-29T10:55:15.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.316+0000 7efd22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:15.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.316+0000 7efd22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:15.297 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:15.297 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:15.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 10 librados: watch_flush enter 2024-03-29T10:55:15.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 10 librados: watch_flush exit 2024-03-29T10:55:15.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T10:55:15.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 10 client.4342.objecter successfully canceled tick 2024-03-29T10:55:15.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 >> v1:172.21.15.183:6816/1412921548 conn(0x7efd1006e000 legacy=0x7efd100704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 10 monclient: shutdown 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 >> v1:172.21.15.183:6789/0 conn(0x7efd2c0b0860 legacy=0x7efd2c1c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.320+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 shutdown_connections 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.321+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 >> 172.21.15.155:0/413262574 conn(0x7efd2c06b660 msgr2=0x7efd2c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.321+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 shutdown_connections 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.321+0000 7efd34372640 1 -- 172.21.15.155:0/413262574 wait complete. 2024-03-29T10:55:15.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:15.321+0000 7efd34372640 1 librados: shutdown 2024-03-29T10:55:15.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:22.125 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.295+0000 7f937bb02640 10 monclient: get_monmap_and_config 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.295+0000 7f937bb02640 10 monclient: build_initial_monmap 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.295+0000 7f937bb02640 10 monclient: monmap: 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:22.295994+0000 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:22.295994+0000 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:22.276 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:22.277 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:22.277 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:22.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.296+0000 7f937bb02640 1 Processor -- start 2024-03-29T10:55:22.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.296+0000 7f937bb02640 1 -- start start 2024-03-29T10:55:22.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.296+0000 7f937bb02640 10 monclient: init 2024-03-29T10:55:22.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.297+0000 7f937bb02640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.297+0000 7f937bb02640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.297+0000 7f937bb02640 10 monclient(hunting): picked mon.noname-a con 0x7f93740b0500 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.297+0000 7f937989e640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f93740b0500 0x7f937413ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:40936/0 (socket says 172.21.15.155:40936) 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937989e640 1 -- 172.21.15.155:0/403449623 learned_addr learned my addr 172.21.15.155:0/403449623 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937bb02640 10 monclient(hunting): picked mon.noname-c con 0x7f93740b08e0 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937bb02640 10 monclient(hunting): picked mon.noname-b con 0x7f93740aeca0 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374140620 con 0x7f93740b0500 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937413daf0 con 0x7f93740b08e0 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93740ae180 con 0x7f93740aeca0 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937bb02640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237270756 0 0) 0x7f9374140620 con 0x7f93740b0500 2024-03-29T10:55:22.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937bb02640 10 monclient(hunting): authenticate will time out at 1025.590669s 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937889c640 10 monclient(hunting): _negotiate 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937889c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937889c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937889c640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.298+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c003520 con 0x7f93740b0500 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548385492 0 0) 0x7f93740ae180 con 0x7f93740aeca0 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient(hunting): _negotiate 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9374140620 con 0x7f93740aeca0 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 877338656 0 0) 0x7f935c003520 con 0x7f93740b0500 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 >> v1:172.21.15.155:6790/0 conn(0x7f93740aeca0 legacy=0x7f937414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:22.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 >> v1:172.21.15.183:6789/0 conn(0x7f93740b08e0 legacy=0x7f937413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient: _finish_hunting 0 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 monclient: found mon.noname-a 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93740accf0 con 0x7f93740b0500 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient: _finish_auth 0 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient: _check_auth_tickets 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9364001690 con 0x7f93740b0500 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937889c640 10 monclient: dump: 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:22.285 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937bb02640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.299+0000 7f937bb02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.300+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93640017f0 con 0x7f93740b0500 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937889c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937889c640 1 -- 172.21.15.155:0/403449623 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9364001c00 con 0x7f93740b0500 2024-03-29T10:55:22.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937889c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937889c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937889c640 10 monclient: dump: 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937bb02640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:22.287 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937bb02640 4 set_mon_vals no callback set 2024-03-29T10:55:22.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.301+0000 7f937bb02640 10 monclient: shutdown 2024-03-29T10:55:22.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.302+0000 7f937bb02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:22.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.302+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 >> v1:172.21.15.155:6789/0 conn(0x7f93740b0500 legacy=0x7f937413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:22.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.302+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 shutdown_connections 2024-03-29T10:55:22.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.302+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 >> 172.21.15.155:0/403449623 conn(0x7f937406b660 msgr2=0x7f937406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:22.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.302+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 shutdown_connections 2024-03-29T10:55:22.288 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.302+0000 7f937bb02640 1 -- 172.21.15.155:0/403449623 wait complete. 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.303+0000 7f937bb02640 10 monclient: build_initial_monmap 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.303+0000 7f937bb02640 10 monclient: monmap: 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:22.292 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.304+0000 7f937bb02640 1 librados: starting msgr at 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.304+0000 7f937bb02640 1 librados: starting objecter 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.304+0000 7f937bb02640 1 Processor -- start 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.304+0000 7f937bb02640 1 -- start start 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.304+0000 7f937bb02640 1 librados: setting wanted keys 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.304+0000 7f937bb02640 1 librados: calling monclient init 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.304+0000 7f937bb02640 10 monclient: init 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 10 monclient(hunting): picked mon.noname-a con 0x7f93740aeca0 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 10 monclient(hunting): picked mon.noname-b con 0x7f93740b0500 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 10 monclient(hunting): picked mon.noname-c con 0x7f937413b880 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:22.293 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93741d9020 con 0x7f93740aeca0 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93741d92e0 con 0x7f93740b0500 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9374121ea0 con 0x7f937413b880 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937bb02640 10 monclient(hunting): authenticate will time out at 1025.597590s 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937989e640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f93740aeca0 0x7f93741d8910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:41718/0 (socket says 172.21.15.155:41718) 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f937989e640 1 -- 172.21.15.155:0/4005420035 learned_addr learned my addr 172.21.15.155:0/4005420035 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f936a7fc640 10 client.?.objecter ms_handle_connect 0x7f93740aeca0 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f936a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f936a7fc640 10 client.?.objecter ms_handle_connect 0x7f937413b880 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.305+0000 7f936a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717899888 0 0) 0x7f9374121ea0 con 0x7f937413b880 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:22.294 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9360004eb0 con 0x7f937413b880 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2577621850 0 0) 0x7f9360004eb0 con 0x7f937413b880 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9374121ea0 con 0x7f937413b880 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 client.?.objecter ms_handle_connect 0x7f93740b0500 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318600731 0 0) 0x7f93741d9020 con 0x7f93740aeca0 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.306+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9360004eb0 con 0x7f93740aeca0 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f936c001dd0 con 0x7f937413b880 2024-03-29T10:55:22.295 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): dump: 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004262309 0 0) 0x7f93741d92e0 con 0x7f93740b0500 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:22.296 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93741d9020 con 0x7f93740b0500 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1887609992 0 0) 0x7f9374121ea0 con 0x7f937413b880 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 >> v1:172.21.15.183:6789/0 conn(0x7f93740b0500 legacy=0x7f937413ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 >> v1:172.21.15.155:6789/0 conn(0x7f93740aeca0 legacy=0x7f93741d8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 monclient: found mon.c 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937413b280 con 0x7f937413b880 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.307+0000 7f936a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 5 monclient: authenticate success, global_id 4370 2024-03-29T10:55:22.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 10 monclient: _renew_subs 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937413b490 con 0x7f937413b880 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f936c003990 con 0x7f937413b880 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f936a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f936c003ca0 con 0x7f937413b880 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f936a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f936a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f936a7fc640 10 monclient: dump: 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:22.298 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f936bfff640 4 set_mon_vals no callback set 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 10 monclient: _renew_subs 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93741203a0 con 0x7f937413b880 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.308+0000 7f937bb02640 1 librados: init done 2024-03-29T10:55:22.299 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.311+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f936c019c70 con 0x7f937413b880 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.312+0000 7f935bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.312+0000 7f935bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.312+0000 7f935bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.312+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9350005080 con 0x7f937413b880 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.313+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f936c083c50 con 0x7f937413b880 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.313+0000 7f936a7fc640 10 client.4370.objecter ms_dispatch 0x7f9374000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.313+0000 7f936a7fc640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.313+0000 7f936a7fc640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.313+0000 7f936a7fc640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.313+0000 7f937989e640 10 monclient: get_auth_request con 0x7f9360072260 auth_method 0 2024-03-29T10:55:22.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.313+0000 7f936a7fc640 10 client.4370.objecter ms_handle_connect 0x7f9360072260 2024-03-29T10:55:22.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.322+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f936c04f530 con 0x7f937413b880 2024-03-29T10:55:22.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.322+0000 7f936a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:22.302 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.322+0000 7f936a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:22.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.706+0000 7f935bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:22.687 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.706+0000 7f935bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:22.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.706+0000 7f935bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:22.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.706+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9350005330 con 0x7f937413b880 2024-03-29T10:55:22.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.706+0000 7f936a7fc640 1 -- 172.21.15.155:0/4005420035 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f936c051460 con 0x7f937413b880 2024-03-29T10:55:22.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.706+0000 7f936a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:22.688 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.706+0000 7f936a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:22.688 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:22.688 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 10 librados: watch_flush enter 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 10 librados: watch_flush exit 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 10 client.4370.objecter successfully canceled tick 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 >> v1:172.21.15.183:6816/1412921548 conn(0x7f9360072260 legacy=0x7f9360074720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 10 monclient: shutdown 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:22.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 >> v1:172.21.15.155:6790/0 conn(0x7f937413b880 legacy=0x7f937411fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:22.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 shutdown_connections 2024-03-29T10:55:22.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.710+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 >> 172.21.15.155:0/4005420035 conn(0x7f937406b660 msgr2=0x7f93740a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:22.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.711+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 shutdown_connections 2024-03-29T10:55:22.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.711+0000 7f935bfff640 1 -- 172.21.15.155:0/4005420035 wait complete. 2024-03-29T10:55:22.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:22.711+0000 7f935bfff640 1 librados: shutdown 2024-03-29T10:55:22.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:29.512 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:29.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.665+0000 7fbbf247b640 10 monclient: get_monmap_and_config 2024-03-29T10:55:29.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.665+0000 7fbbf247b640 10 monclient: build_initial_monmap 2024-03-29T10:55:29.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.665+0000 7fbbf247b640 10 monclient: monmap: 2024-03-29T10:55:29.647 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:29.647 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:29.667265+0000 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:29.667265+0000 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:29.648 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.666+0000 7fbbf247b640 1 Processor -- start 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.666+0000 7fbbf247b640 1 -- start start 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.666+0000 7fbbf247b640 10 monclient: init 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.667+0000 7fbbf247b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.667+0000 7fbbf247b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.667+0000 7fbbf247b640 10 monclient(hunting): picked mon.noname-a con 0x7fbbec1457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.667+0000 7fbbf247b640 10 monclient(hunting): picked mon.noname-c con 0x7fbbec13f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf247b640 10 monclient(hunting): picked mon.noname-b con 0x7fbbec0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:29.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf247b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec0aaef0 con 0x7fbbec1457f0 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf247b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec145bd0 con 0x7fbbec13f650 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf247b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec145eb0 con 0x7fbbec0a9d70 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf247b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf247b640 10 monclient(hunting): authenticate will time out at 1032.961035s 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf0a18640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fbbec0a9d70 0x7fbbec14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:32832/0 (socket says 172.21.15.155:32832) 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbf0a18640 1 -- 172.21.15.155:0/442092250 learned_addr learned my addr 172.21.15.155:0/442092250 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919132917 0 0) 0x7fbbec145eb0 con 0x7fbbec0a9d70 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:55:29.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd4003520 con 0x7fbbec0a9d70 2024-03-29T10:55:29.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274375921 0 0) 0x7fbbd4003520 con 0x7fbbec0a9d70 2024-03-29T10:55:29.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.668+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 >> v1:172.21.15.183:6789/0 conn(0x7fbbec13f650 legacy=0x7fbbec0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:29.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.669+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 >> v1:172.21.15.155:6789/0 conn(0x7fbbec1457f0 legacy=0x7fbbec13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 1 monclient: found mon.noname-b 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec146340 con 0x7fbbec0a9d70 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbf247b640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:55:29.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbf247b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbbe4001a10 con 0x7fbbec0a9d70 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbeaffd640 10 monclient: dump: 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:29.656 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.671+0000 7fbbf247b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe4001ca0 con 0x7fbbec0a9d70 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbeaffd640 1 -- 172.21.15.155:0/442092250 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbbe4002030 con 0x7fbbec0a9d70 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbf247b640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbf247b640 4 set_mon_vals no callback set 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.672+0000 7fbbeaffd640 10 monclient: dump: 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.657 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.673+0000 7fbbf247b640 10 monclient: shutdown 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.673+0000 7fbbf247b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.673+0000 7fbbf247b640 1 -- 172.21.15.155:0/442092250 >> v1:172.21.15.155:6790/0 conn(0x7fbbec0a9d70 legacy=0x7fbbec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.673+0000 7fbbf247b640 1 -- 172.21.15.155:0/442092250 shutdown_connections 2024-03-29T10:55:29.658 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.673+0000 7fbbf247b640 1 -- 172.21.15.155:0/442092250 >> 172.21.15.155:0/442092250 conn(0x7fbbec066a00 msgr2=0x7fbbec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:29.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.674+0000 7fbbf247b640 1 -- 172.21.15.155:0/442092250 shutdown_connections 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.674+0000 7fbbf247b640 1 -- 172.21.15.155:0/442092250 wait complete. 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.675+0000 7fbbf247b640 10 monclient: build_initial_monmap 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.675+0000 7fbbf247b640 10 monclient: monmap: 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:29.661 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.676+0000 7fbbf247b640 1 librados: starting msgr at 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.676+0000 7fbbf247b640 1 librados: starting objecter 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.676+0000 7fbbf247b640 1 Processor -- start 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.676+0000 7fbbf247b640 1 -- start start 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.676+0000 7fbbf247b640 1 librados: setting wanted keys 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.676+0000 7fbbf247b640 1 librados: calling monclient init 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.676+0000 7fbbf247b640 10 monclient: init 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.677+0000 7fbbf247b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:29.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.677+0000 7fbbf247b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 10 monclient(hunting): picked mon.noname-a con 0x7fbbec0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbeb7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fbbec13f650 0x7fbbec1d7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34886/0 (socket says 172.21.15.155:34886) 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbebfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbbec0a9d70 0x7fbbec1361b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:47661/0 (socket says 172.21.15.155:47661) 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbeb7fe640 1 -- 172.21.15.155:0/1432580646 learned_addr learned my addr 172.21.15.155:0/1432580646 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 10 monclient(hunting): picked mon.noname-c con 0x7fbbec13f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbe8ff9640 10 client.?.objecter ms_handle_connect 0x7fbbec0a9d70 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbe8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:29.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbe8ff9640 10 client.?.objecter ms_handle_connect 0x7fbbec13f650 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbe8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 10 monclient(hunting): picked mon.noname-b con 0x7fbbec1457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec134300 con 0x7fbbec0a9d70 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec133730 con 0x7fbbec1457f0 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec1338c0 con 0x7fbbec13f650 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.681+0000 7fbbf247b640 10 monclient(hunting): authenticate will time out at 1032.974844s 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583073163 0 0) 0x7fbbec1338c0 con 0x7fbbec13f650 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8004eb0 con 0x7fbbec13f650 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1319155028 0 0) 0x7fbbd8004eb0 con 0x7fbbec13f650 2024-03-29T10:55:29.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbec1338c0 con 0x7fbbec13f650 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbbe0001870 con 0x7fbbec13f650 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:29.666 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:29.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:29.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610967795 0 0) 0x7fbbec134300 con 0x7fbbec0a9d70 2024-03-29T10:55:29.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:29.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:29.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:29.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:55:29.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.682+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8004eb0 con 0x7fbbec0a9d70 2024-03-29T10:55:29.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.683+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2547392991 0 0) 0x7fbbec1338c0 con 0x7fbbec13f650 2024-03-29T10:55:29.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.683+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 >> v1:172.21.15.183:6789/0 conn(0x7fbbec1457f0 legacy=0x7fbbec1e1ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:29.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 >> v1:172.21.15.155:6789/0 conn(0x7fbbec0a9d70 legacy=0x7fbbec1361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:29.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:29.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 1 monclient: found mon.c 2024-03-29T10:55:29.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec133a50 con 0x7fbbec13f650 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 5 monclient: authenticate success, global_id 4376 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 10 monclient: _renew_subs 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe0001e00 con 0x7fbbec13f650 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec1424c0 con 0x7fbbec13f650 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbbe0004aa0 con 0x7fbbec13f650 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:29.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbe8ff9640 10 monclient: dump: 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 10 monclient: _renew_subs 2024-03-29T10:55:29.670 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec134790 con 0x7fbbec13f650 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.684+0000 7fbbf247b640 1 librados: init done 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.685+0000 7fbbea7fc640 4 set_mon_vals no callback set 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.686+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fbbe001aa70 con 0x7fbbec13f650 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.686+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fbbe0084b30 con 0x7fbbec13f650 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.686+0000 7fbbe8ff9640 10 client.4376.objecter ms_dispatch 0x7fbbec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.686+0000 7fbbe8ff9640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.686+0000 7fbbe8ff9640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:29.671 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.686+0000 7fbbe8ff9640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T10:55:29.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.687+0000 7fbbebfff640 10 monclient: get_auth_request con 0x7fbbd806dd70 auth_method 0 2024-03-29T10:55:29.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.687+0000 7fbbe8ff9640 10 client.4376.objecter ms_handle_connect 0x7fbbd806dd70 2024-03-29T10:55:29.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.687+0000 7fbbf247b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:29.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.687+0000 7fbbf247b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:29.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.687+0000 7fbbf247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:29.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.687+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbcc005080 con 0x7fbbec13f650 2024-03-29T10:55:29.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.696+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbe0050240 con 0x7fbbec13f650 2024-03-29T10:55:29.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.696+0000 7fbbe8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:29.678 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:29.696+0000 7fbbe8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:30.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.022+0000 7fbbf247b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:30.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.022+0000 7fbbf247b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:30.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.022+0000 7fbbf247b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:30.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.022+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbcc005330 con 0x7fbbec13f650 2024-03-29T10:55:30.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.023+0000 7fbbe8ff9640 1 -- 172.21.15.155:0/1432580646 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fbbe0052340 con 0x7fbbec13f650 2024-03-29T10:55:30.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.023+0000 7fbbe8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:30.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.023+0000 7fbbe8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:30.005 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:30.005 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 10 librados: watch_flush enter 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 10 librados: watch_flush exit 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 10 client.4376.objecter successfully canceled tick 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 >> v1:172.21.15.183:6816/1412921548 conn(0x7fbbd806dd70 legacy=0x7fbbd8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 10 monclient: shutdown 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:30.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 >> v1:172.21.15.155:6790/0 conn(0x7fbbec13f650 legacy=0x7fbbec1d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:30.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 shutdown_connections 2024-03-29T10:55:30.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.027+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 >> 172.21.15.155:0/1432580646 conn(0x7fbbec066a00 msgr2=0x7fbbec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:30.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.028+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 shutdown_connections 2024-03-29T10:55:30.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.028+0000 7fbbf247b640 1 -- 172.21.15.155:0/1432580646 wait complete. 2024-03-29T10:55:30.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:30.028+0000 7fbbf247b640 1 librados: shutdown 2024-03-29T10:55:30.027 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:36.834 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:36.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.001+0000 7f188e90e640 10 monclient: get_monmap_and_config 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.001+0000 7f188e90e640 10 monclient: build_initial_monmap 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.001+0000 7f188e90e640 10 monclient: monmap: 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:37.003051+0000 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:37.003051+0000 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:36.983 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:36.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.002+0000 7f188e90e640 1 Processor -- start 2024-03-29T10:55:36.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.002+0000 7f188e90e640 1 -- start start 2024-03-29T10:55:36.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.002+0000 7f188e90e640 10 monclient: init 2024-03-29T10:55:36.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 10 monclient(hunting): picked mon.noname-a con 0x7f18881457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 10 monclient(hunting): picked mon.noname-b con 0x7f188813f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 10 monclient(hunting): picked mon.noname-c con 0x7f18880a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18880aaef0 con 0x7f18881457f0 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1888145bd0 con 0x7f18880a9d70 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1888145eb0 con 0x7f188813f650 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:36.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.003+0000 7f188e90e640 10 monclient(hunting): authenticate will time out at 1040.296324s 2024-03-29T10:55:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.006+0000 7f188ceab640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f18880a9d70 0x7f188814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:59116/0 (socket says 172.21.15.155:59116) 2024-03-29T10:55:36.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.006+0000 7f188ceab640 1 -- 172.21.15.155:0/1630352949 learned_addr learned my addr 172.21.15.155:0/1630352949 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:36.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646778772 0 0) 0x7f18880aaef0 con 0x7f18881457f0 2024-03-29T10:55:36.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:36.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:36.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:36.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:55:36.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1870003520 con 0x7f18881457f0 2024-03-29T10:55:36.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3115745487 0 0) 0x7f1870003520 con 0x7f18881457f0 2024-03-29T10:55:36.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.008+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 >> v1:172.21.15.155:6790/0 conn(0x7f188813f650 legacy=0x7f18880b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:36.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 >> v1:172.21.15.183:6789/0 conn(0x7f18880a9d70 legacy=0x7f188814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 1 monclient: found mon.noname-a 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1888146340 con 0x7f18881457f0 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f188e90e640 5 monclient: authenticate success, global_id 4356 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f188e90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1880001a10 con 0x7f18881457f0 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.009+0000 7f1886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.010+0000 7f1886ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.010+0000 7f1886ffd640 10 monclient: dump: 2024-03-29T10:55:36.993 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.010+0000 7f188e90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.010+0000 7f1886ffd640 1 -- 172.21.15.155:0/1630352949 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1880001ca0 con 0x7f18881457f0 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.010+0000 7f1886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.010+0000 7f188e90e640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:36.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.010+0000 7f188e90e640 4 set_mon_vals no callback set 2024-03-29T10:55:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.011+0000 7f188e90e640 10 monclient: shutdown 2024-03-29T10:55:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.011+0000 7f188e90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.011+0000 7f188e90e640 1 -- 172.21.15.155:0/1630352949 >> v1:172.21.15.155:6789/0 conn(0x7f18881457f0 legacy=0x7f188813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.011+0000 7f188e90e640 1 -- 172.21.15.155:0/1630352949 shutdown_connections 2024-03-29T10:55:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.011+0000 7f188e90e640 1 -- 172.21.15.155:0/1630352949 >> 172.21.15.155:0/1630352949 conn(0x7f1888066a00 msgr2=0x7f1888066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.011+0000 7f188e90e640 1 -- 172.21.15.155:0/1630352949 shutdown_connections 2024-03-29T10:55:36.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.011+0000 7f188e90e640 1 -- 172.21.15.155:0/1630352949 wait complete. 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.012+0000 7f188e90e640 10 monclient: build_initial_monmap 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.012+0000 7f188e90e640 10 monclient: monmap: 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.012+0000 7f188e90e640 1 librados: starting msgr at 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.012+0000 7f188e90e640 1 librados: starting objecter 2024-03-29T10:55:36.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.013+0000 7f188e90e640 1 Processor -- start 2024-03-29T10:55:36.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.013+0000 7f188e90e640 1 -- start start 2024-03-29T10:55:36.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.013+0000 7f188e90e640 1 librados: setting wanted keys 2024-03-29T10:55:36.997 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.013+0000 7f188e90e640 1 librados: calling monclient init 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.013+0000 7f188e90e640 10 monclient: init 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 10 monclient(hunting): picked mon.noname-b con 0x7f18880a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 10 monclient(hunting): picked mon.noname-c con 0x7f188813f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 10 monclient(hunting): picked mon.noname-a con 0x7f18881457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f188806e4c0 con 0x7f18881457f0 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18880b7940 con 0x7f18880a9d70 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18880b7540 con 0x7f188813f650 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.014+0000 7f188e90e640 10 monclient(hunting): authenticate will time out at 1040.307026s 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.016+0000 7f1887fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f18880a9d70 0x7f18880b6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:60494/0 (socket says 172.21.15.155:60494) 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.016+0000 7f1887fff640 1 -- 172.21.15.155:0/3789695229 learned_addr learned my addr 172.21.15.155:0/3789695229 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:36.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.016+0000 7f1884ff9640 10 client.?.objecter ms_handle_connect 0x7f188813f650 2024-03-29T10:55:36.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.016+0000 7f1884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.016+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014707115 0 0) 0x7f18880b7540 con 0x7f188813f650 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.016+0000 7f1884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1874004eb0 con 0x7f188813f650 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 client.?.objecter ms_handle_connect 0x7f18880a9d70 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:37.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936115434 0 0) 0x7f18880b7940 con 0x7f18880a9d70 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18880b7540 con 0x7f18880a9d70 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1259717633 0 0) 0x7f1874004eb0 con 0x7f188813f650 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18880b7940 con 0x7f188813f650 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f187c001870 con 0x7f188813f650 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.017+0000 7f1884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.018+0000 7f1884ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.018+0000 7f1884ff9640 10 monclient(hunting): dump: 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:37.002 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.018+0000 7f1884ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.018+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3724992340 0 0) 0x7f18880b7940 con 0x7f188813f650 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.018+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.183:6789/0 conn(0x7f18880a9d70 legacy=0x7f18880b6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.018+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6789/0 conn(0x7f18881457f0 legacy=0x7f18881d6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 1 monclient: found mon.c 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18880b76d0 con 0x7f188813f650 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 10 monclient: _renew_subs 2024-03-29T10:55:37.005 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18881374b0 con 0x7f188813f650 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f187c0019d0 con 0x7f188813f650 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f1884ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 10 monclient: _renew_subs 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1888141e70 con 0x7f188813f650 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f188e90e640 1 librados: init done 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.020+0000 7f18867fc640 4 set_mon_vals no callback set 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.021+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f187c0037d0 con 0x7f188813f650 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.021+0000 7f1884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.021+0000 7f1884ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.021+0000 7f1884ff9640 10 monclient: dump: 2024-03-29T10:55:37.006 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:37.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.023+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f187c0197a0 con 0x7f188813f650 2024-03-29T10:55:37.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.023+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f187c083830 con 0x7f188813f650 2024-03-29T10:55:37.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.023+0000 7f1884ff9640 10 client.4379.objecter ms_dispatch 0x7f1888000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:37.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.023+0000 7f1884ff9640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:37.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.023+0000 7f1884ff9640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.023+0000 7f1884ff9640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f1887fff640 10 monclient: get_auth_request con 0x7f187406ddd0 auth_method 0 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f188e90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f188e90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f188e90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18880b7b80 con 0x7f188813f650 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f18877fe640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6790/0 conn(0x7f188813f650 legacy=0x7f1888136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f18877fe640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6790/0 conn(0x7f188813f650 legacy=0x7f1888136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f18877fe640 1 --1- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6790/0 conn(0x7f188813f650 0x7f1888136da0 :-1 s=OPENED pgs=353415 cs=1 l=1).handle_message read tag failed 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f18877fe640 1 --1- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6790/0 conn(0x7f188813f650 0x7f1888136da0 :-1 s=OPENED pgs=353415 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f1884ff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.155:6790/0 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f1884ff9640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6790/0 conn(0x7f188813f650 legacy=0x7f1888136da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.024+0000 7f1884ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:37.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.025+0000 7f1884ff9640 10 monclient(hunting): picked mon.a con 0x7f18881457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:37.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.025+0000 7f1884ff9640 10 monclient(hunting): picked mon.b con 0x7f18880a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:37.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.026+0000 7f1884ff9640 10 monclient(hunting): picked mon.c con 0x7f187407a1b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:37.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.026+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f18880b7940 con 0x7f18881457f0 2024-03-29T10:55:37.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.026+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f188806e4c0 con 0x7f18880a9d70 2024-03-29T10:55:37.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.026+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1874004eb0 con 0x7f187407a1b0 2024-03-29T10:55:37.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.026+0000 7f1884ff9640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:37.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.026+0000 7f1884ff9640 10 client.4379.objecter ms_handle_connect 0x7f187406ddd0 2024-03-29T10:55:37.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 10 client.4379.objecter ms_handle_connect 0x7f18881457f0 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 10 client.4379.objecter resend_mon_ops 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401757024 0 0) 0x7f18880b7940 con 0x7f18881457f0 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f187407e290 con 0x7f18881457f0 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.028+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 4150903617 0 0) 0x7f187407e290 con 0x7f18881457f0 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6790/0 conn(0x7f187407a1b0 legacy=0x7f187407c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.183:6789/0 conn(0x7f18880a9d70 legacy=0x7f1874078590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 1 monclient: found mon.a 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:37.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f18740813c0 con 0x7f18881457f0 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f188806e4c0 con 0x7f18881457f0 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.029+0000 7f1884ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.030+0000 7f1884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:37.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.030+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f187c052cd0 con 0x7f18881457f0 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.031+0000 7f1884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.031+0000 7f1884ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.031+0000 7f1884ff9640 10 monclient: dump: 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:37.014 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:37.015 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:37.015 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:37.015 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:37.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.031+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f187c052860 con 0x7f18881457f0 2024-03-29T10:55:37.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.031+0000 7f1884ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:37.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.031+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f187c04ebc0 con 0x7f18881457f0 2024-03-29T10:55:37.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.032+0000 7f1885ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:37.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.040+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f187c07afd0 con 0x7f18881457f0 2024-03-29T10:55:37.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.040+0000 7f1884ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:37.022 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.040+0000 7f1884ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:37.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.372+0000 7f188e90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:37.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.372+0000 7f188e90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:37.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.372+0000 7f188e90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:55:37.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.372+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18880b7b80 con 0x7f18881457f0 2024-03-29T10:55:37.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.375+0000 7f1884ff9640 1 -- 172.21.15.155:0/3789695229 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f18880b7b80 con 0x7f18881457f0 2024-03-29T10:55:37.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.375+0000 7f1884ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:37.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.375+0000 7f1884ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:37.357 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:37.357 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:37.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.379+0000 7f188e90e640 10 librados: watch_flush enter 2024-03-29T10:55:37.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.379+0000 7f188e90e640 10 librados: watch_flush exit 2024-03-29T10:55:37.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.379+0000 7f188e90e640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.379+0000 7f188e90e640 10 client.4379.objecter successfully canceled tick 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.379+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.183:6816/1412921548 conn(0x7f187406ddd0 legacy=0x7f1874070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.380+0000 7f188e90e640 10 monclient: shutdown 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.380+0000 7f188e90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.380+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 >> v1:172.21.15.155:6789/0 conn(0x7f18881457f0 legacy=0x7f1874074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.380+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 shutdown_connections 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.380+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 >> 172.21.15.155:0/3789695229 conn(0x7f1888066a00 msgr2=0x7f188813cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.380+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 shutdown_connections 2024-03-29T10:55:37.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.380+0000 7f188e90e640 1 -- 172.21.15.155:0/3789695229 wait complete. 2024-03-29T10:55:37.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:37.383+0000 7f188e90e640 1 librados: shutdown 2024-03-29T10:55:37.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:44.188 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.363+0000 7f0f48fb4640 10 monclient: get_monmap_and_config 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.363+0000 7f0f48fb4640 10 monclient: build_initial_monmap 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.363+0000 7f0f48fb4640 10 monclient: monmap: 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:44.365061+0000 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:44.365061+0000 2024-03-29T10:55:44.345 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:44.346 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:44.346 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:44.346 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:44.346 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:44.346 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:44.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.365+0000 7f0f48fb4640 1 Processor -- start 2024-03-29T10:55:44.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.365+0000 7f0f48fb4640 1 -- start start 2024-03-29T10:55:44.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.365+0000 7f0f48fb4640 10 monclient: init 2024-03-29T10:55:44.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 10 monclient(hunting): picked mon.noname-c con 0x7f0f3c006eb0 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 10 monclient(hunting): picked mon.noname-b con 0x7f0f3c038f10 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 10 monclient(hunting): picked mon.noname-a con 0x7f0f3c036690 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c033c90 con 0x7f0f3c036690 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c0389f0 con 0x7f0f3c006eb0 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c030890 con 0x7f0f3c038f10 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.366+0000 7f0f48fb4640 10 monclient(hunting): authenticate will time out at 1047.659642s 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.369+0000 7f0f437fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f0f3c006eb0 0x7f0f3c031760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:56559/0 (socket says 172.21.15.155:56559) 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.369+0000 7f0f437fe640 1 -- 172.21.15.155:0/2825998298 learned_addr learned my addr 172.21.15.155:0/2825998298 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:44.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163122909 0 0) 0x7f0f3c0389f0 con 0x7f0f3c006eb0 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f34003520 con 0x7f0f3c006eb0 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2556733323 0 0) 0x7f0f34003520 con 0x7f0f3c006eb0 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.370+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 >> v1:172.21.15.155:6790/0 conn(0x7f0f3c038f10 legacy=0x7f0f3c039330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:44.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.372+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 >> v1:172.21.15.155:6789/0 conn(0x7f0f3c036690 legacy=0x7f0f3c036b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 1 monclient: found mon.noname-c 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c0cbc40 con 0x7f0f3c006eb0 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f48fb4640 5 monclient: authenticate success, global_id 4351 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f48fb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0f4404f730 con 0x7f0f3c006eb0 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:44.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f427fc640 10 monclient: dump: 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:44.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.373+0000 7f0f48fb4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.376+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f44075f10 con 0x7f0f3c006eb0 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.376+0000 7f0f427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.376+0000 7f0f48fb4640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.376+0000 7f0f48fb4640 4 set_mon_vals no callback set 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.377+0000 7f0f427fc640 1 -- 172.21.15.155:0/2825998298 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0f440758b0 con 0x7f0f3c006eb0 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.377+0000 7f0f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.377+0000 7f0f427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:44.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.377+0000 7f0f427fc640 10 monclient: dump: 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:44.359 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:44.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.377+0000 7f0f48fb4640 10 monclient: shutdown 2024-03-29T10:55:44.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.377+0000 7f0f48fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:44.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.377+0000 7f0f48fb4640 1 -- 172.21.15.155:0/2825998298 >> v1:172.21.15.183:6789/0 conn(0x7f0f3c006eb0 legacy=0x7f0f3c031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:44.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.380+0000 7f0f48fb4640 1 -- 172.21.15.155:0/2825998298 shutdown_connections 2024-03-29T10:55:44.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.380+0000 7f0f48fb4640 1 -- 172.21.15.155:0/2825998298 >> 172.21.15.155:0/2825998298 conn(0x7f0f3c005c20 msgr2=0x7f0f3c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:44.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.381+0000 7f0f48fb4640 1 -- 172.21.15.155:0/2825998298 shutdown_connections 2024-03-29T10:55:44.363 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.381+0000 7f0f48fb4640 1 -- 172.21.15.155:0/2825998298 wait complete. 2024-03-29T10:55:44.364 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.382+0000 7f0f48fb4640 10 monclient: build_initial_monmap 2024-03-29T10:55:44.369 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.382+0000 7f0f48fb4640 10 monclient: monmap: 2024-03-29T10:55:44.369 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:44.369 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:44.369 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:44.369 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.383+0000 7f0f48fb4640 1 librados: starting msgr at 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.383+0000 7f0f48fb4640 1 librados: starting objecter 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.383+0000 7f0f48fb4640 1 Processor -- start 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.383+0000 7f0f48fb4640 1 -- start start 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.383+0000 7f0f48fb4640 1 librados: setting wanted keys 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.383+0000 7f0f48fb4640 1 librados: calling monclient init 2024-03-29T10:55:44.370 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.383+0000 7f0f48fb4640 10 monclient: init 2024-03-29T10:55:44.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:44.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:44.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 10 monclient(hunting): picked mon.noname-a con 0x7f0f3c006eb0 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:44.371 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 10 monclient(hunting): picked mon.noname-c con 0x7f0f3c036690 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:44.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 10 monclient(hunting): picked mon.noname-b con 0x7f0f3c038f10 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:44.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c0cfca0 con 0x7f0f3c006eb0 2024-03-29T10:55:44.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c00b470 con 0x7f0f3c038f10 2024-03-29T10:55:44.374 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c00a400 con 0x7f0f3c036690 2024-03-29T10:55:44.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:44.377 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.384+0000 7f0f48fb4640 10 monclient(hunting): authenticate will time out at 1047.677421s 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.393+0000 7f0f437fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f0f3c006eb0 0x7f0f3c0c8aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:60517/0 (socket says 172.21.15.155:60517) 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.393+0000 7f0f437fe640 1 -- 172.21.15.155:0/844434322 learned_addr learned my addr 172.21.15.155:0/844434322 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.393+0000 7f0f2ffff640 10 client.?.objecter ms_handle_connect 0x7f0f3c038f10 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.393+0000 7f0f2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.393+0000 7f0f2ffff640 10 client.?.objecter ms_handle_connect 0x7f0f3c036690 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510176884 0 0) 0x7f0f3c00b470 con 0x7f0f3c038f10 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:44.383 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f28004eb0 con 0x7f0f3c038f10 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 client.?.objecter ms_handle_connect 0x7f0f3c006eb0 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191505526 0 0) 0x7f0f3c00a400 con 0x7f0f3c036690 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3c00b470 con 0x7f0f3c036690 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1123482936 0 0) 0x7f0f28004eb0 con 0x7f0f3c038f10 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f3c00a400 con 0x7f0f3c038f10 2024-03-29T10:55:44.384 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0f380017f0 con 0x7f0f3c038f10 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): dump: 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:44.385 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3149088370 0 0) 0x7f0f3c00b470 con 0x7f0f3c036690 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f28004eb0 con 0x7f0f3c036690 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0f30001790 con 0x7f0f3c036690 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient(hunting): dump: 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:44.386 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4250617104 0 0) 0x7f0f3c00a400 con 0x7f0f3c038f10 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 >> v1:172.21.15.155:6790/0 conn(0x7f0f3c036690 legacy=0x7f0f3c15db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 >> v1:172.21.15.155:6789/0 conn(0x7f0f3c006eb0 legacy=0x7f0f3c0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 monclient: found mon.b 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c00a590 con 0x7f0f3c038f10 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient: _finish_auth 0 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:44.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 5 monclient: authenticate success, global_id 4354 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 10 monclient: _renew_subs 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c0c91b0 con 0x7f0f3c038f10 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 10 monclient: _renew_subs 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c0c9550 con 0x7f0f3c038f10 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.394+0000 7f0f48fb4640 1 librados: init done 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.395+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f38001950 con 0x7f0f3c038f10 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.395+0000 7f0f2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.395+0000 7f0f417fa640 4 set_mon_vals no callback set 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.395+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0f38003790 con 0x7f0f3c038f10 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.395+0000 7f0f2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:44.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.395+0000 7f0f2ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.395+0000 7f0f2ffff640 10 monclient: dump: 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.396+0000 7f0f48fb4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:44.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.396+0000 7f0f48fb4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.396+0000 7f0f48fb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.396+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f24005080 con 0x7f0f3c038f10 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.397+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f0f380038f0 con 0x7f0f3c038f10 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.398+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f0f38083750 con 0x7f0f3c038f10 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.398+0000 7f0f2ffff640 10 client.4354.objecter ms_dispatch 0x7f0f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.398+0000 7f0f2ffff640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.398+0000 7f0f2ffff640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.398+0000 7f0f2ffff640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.398+0000 7f0f437fe640 10 monclient: get_auth_request con 0x7f0f2806de90 auth_method 0 2024-03-29T10:55:44.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.398+0000 7f0f2ffff640 10 client.4354.objecter ms_handle_connect 0x7f0f2806de90 2024-03-29T10:55:44.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.405+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f3804f030 con 0x7f0f3c038f10 2024-03-29T10:55:44.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.405+0000 7f0f2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:44.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.405+0000 7f0f2ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:44.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.725+0000 7f0f48fb4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:44.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.725+0000 7f0f48fb4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:44.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.726+0000 7f0f48fb4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:44.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.726+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f24005330 con 0x7f0f3c038f10 2024-03-29T10:55:44.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.727+0000 7f0f2ffff640 1 -- 172.21.15.155:0/844434322 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f0f38050f60 con 0x7f0f3c038f10 2024-03-29T10:55:44.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.728+0000 7f0f2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:44.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.728+0000 7f0f2ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:44.710 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:44.710 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:44.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 10 librados: watch_flush enter 2024-03-29T10:55:44.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 10 librados: watch_flush exit 2024-03-29T10:55:44.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T10:55:44.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 10 client.4354.objecter successfully canceled tick 2024-03-29T10:55:44.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 >> v1:172.21.15.183:6816/1412921548 conn(0x7f0f2806de90 legacy=0x7f0f28070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:44.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 10 monclient: shutdown 2024-03-29T10:55:44.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:44.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.732+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 >> v1:172.21.15.183:6789/0 conn(0x7f0f3c038f10 legacy=0x7f0f3c00cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:44.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.733+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 shutdown_connections 2024-03-29T10:55:44.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.733+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 >> 172.21.15.155:0/844434322 conn(0x7f0f3c005c20 msgr2=0x7f0f3c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:44.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.734+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 shutdown_connections 2024-03-29T10:55:44.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.734+0000 7f0f48fb4640 1 -- 172.21.15.155:0/844434322 wait complete. 2024-03-29T10:55:44.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:44.734+0000 7f0f48fb4640 1 librados: shutdown 2024-03-29T10:55:44.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:51.544 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.712+0000 7fcee2da3640 10 monclient: get_monmap_and_config 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.712+0000 7fcee2da3640 10 monclient: build_initial_monmap 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.712+0000 7fcee2da3640 10 monclient: monmap: 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:51.713107+0000 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:51.713107+0000 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:51.693 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:51.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.713+0000 7fcee2da3640 1 Processor -- start 2024-03-29T10:55:51.694 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.713+0000 7fcee2da3640 1 -- start start 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.713+0000 7fcee2da3640 10 monclient: init 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.714+0000 7fcee2da3640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.714+0000 7fcee2da3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.714+0000 7fcee2da3640 10 monclient(hunting): picked mon.noname-a con 0x7fcedc0b0770 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.714+0000 7fcee2da3640 10 monclient(hunting): picked mon.noname-c con 0x7fcedc14a700 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.717+0000 7fcee0b3f640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fcedc0b0770 0x7fcedc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:58139/0 (socket says 172.21.15.155:58139) 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.717+0000 7fcee2da3640 10 monclient(hunting): picked mon.noname-b con 0x7fcedc0ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.717+0000 7fcee2da3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc0aba70 con 0x7fcedc0b0770 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.717+0000 7fcee2da3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc0aecb0 con 0x7fcedc14a700 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.717+0000 7fcee2da3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc0af150 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.698 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.717+0000 7fcee2da3640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:51.699 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.717+0000 7fcee2da3640 10 monclient(hunting): authenticate will time out at 1055.009085s 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcee0b3f640 1 -- 172.21.15.155:0/79759005 learned_addr learned my addr 172.21.15.155:0/79759005 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231412772 0 0) 0x7fcedc0af150 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc003520 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674421695 0 0) 0x7fcecc003520 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.721+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 >> v1:172.21.15.183:6789/0 conn(0x7fcedc14a700 legacy=0x7fcedc0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 >> v1:172.21.15.155:6789/0 conn(0x7fcedc0b0770 legacy=0x7fcedc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 1 monclient: found mon.noname-b 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc0ac9d0 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:51.704 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fced4001a10 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcedb7fe640 10 monclient: dump: 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcee2da3640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:55:51.705 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.722+0000 7fcee2da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:51.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fced4001be0 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcedb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcedb7fe640 1 -- 172.21.15.155:0/79759005 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fced4001ff0 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcedb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcedb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcedb7fe640 10 monclient: dump: 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:51.708 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:51.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcee2da3640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:51.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.727+0000 7fcee2da3640 4 set_mon_vals no callback set 2024-03-29T10:55:51.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.728+0000 7fcee2da3640 10 monclient: shutdown 2024-03-29T10:55:51.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.728+0000 7fcee2da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:51.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.728+0000 7fcee2da3640 1 -- 172.21.15.155:0/79759005 >> v1:172.21.15.155:6790/0 conn(0x7fcedc0ac5a0 legacy=0x7fcedc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:51.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.728+0000 7fcee2da3640 1 -- 172.21.15.155:0/79759005 shutdown_connections 2024-03-29T10:55:51.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.728+0000 7fcee2da3640 1 -- 172.21.15.155:0/79759005 >> 172.21.15.155:0/79759005 conn(0x7fcedc06b660 msgr2=0x7fcedc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:51.712 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.732+0000 7fcee2da3640 1 -- 172.21.15.155:0/79759005 shutdown_connections 2024-03-29T10:55:51.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.733+0000 7fcee2da3640 1 -- 172.21.15.155:0/79759005 wait complete. 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.734+0000 7fcee2da3640 10 monclient: build_initial_monmap 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.734+0000 7fcee2da3640 10 monclient: monmap: 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:51.715 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:51.716 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:51.716 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:51.716 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:51.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.734+0000 7fcee2da3640 1 librados: starting msgr at 2024-03-29T10:55:51.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.734+0000 7fcee2da3640 1 librados: starting objecter 2024-03-29T10:55:51.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.735+0000 7fcee2da3640 1 Processor -- start 2024-03-29T10:55:51.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.735+0000 7fcee2da3640 1 -- start start 2024-03-29T10:55:51.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.735+0000 7fcee2da3640 1 librados: setting wanted keys 2024-03-29T10:55:51.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.735+0000 7fcee2da3640 1 librados: calling monclient init 2024-03-29T10:55:51.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.735+0000 7fcee2da3640 10 monclient: init 2024-03-29T10:55:51.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.736+0000 7fcee2da3640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:51.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.736+0000 7fcee2da3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:51.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.736+0000 7fcee2da3640 10 monclient(hunting): picked mon.noname-a con 0x7fcedc0ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:51.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.742+0000 7fcee2da3640 10 monclient(hunting): picked mon.noname-b con 0x7fcedc0b0770 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:51.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.742+0000 7fcee2da3640 10 monclient(hunting): picked mon.noname-c con 0x7fcedc14a700 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:51.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.742+0000 7fcee2da3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc128150 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.742+0000 7fcee2da3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc1df480 con 0x7fcedc0b0770 2024-03-29T10:55:51.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.742+0000 7fcee2da3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc145800 con 0x7fcedc14a700 2024-03-29T10:55:51.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.742+0000 7fcee2da3640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:51.723 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.742+0000 7fcee2da3640 10 monclient(hunting): authenticate will time out at 1055.034857s 2024-03-29T10:55:51.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.746+0000 7fcee1340640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fcedc14a700 0x7fcedc1dba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36193/0 (socket says 172.21.15.155:36193) 2024-03-29T10:55:51.727 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.746+0000 7fcee1340640 1 -- 172.21.15.155:0/2394287606 learned_addr learned my addr 172.21.15.155:0/2394287606 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.747+0000 7fced97fa640 10 client.?.objecter ms_handle_connect 0x7fcedc14a700 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.747+0000 7fced97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.747+0000 7fced97fa640 10 client.?.objecter ms_handle_connect 0x7fcedc0ac5a0 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.747+0000 7fced97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.747+0000 7fced97fa640 10 client.?.objecter ms_handle_connect 0x7fcedc0b0770 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.747+0000 7fced97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.747+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033673663 0 0) 0x7fcedc145800 con 0x7fcedc14a700 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:51.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec0004eb0 con 0x7fcedc14a700 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803304835 0 0) 0x7fcedc128150 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcedc145800 con 0x7fcedc0ac5a0 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79353784 0 0) 0x7fcedc1df480 con 0x7fcedc0b0770 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:51.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcedc128150 con 0x7fcedc0b0770 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.748+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388904629 0 0) 0x7fcec0004eb0 con 0x7fcedc14a700 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcedc1df480 con 0x7fcedc14a700 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fced4001ef0 con 0x7fcedc14a700 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 10 monclient(hunting): dump: 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:51.735 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687188673 0 0) 0x7fcedc128150 con 0x7fcedc0b0770 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcec0004eb0 con 0x7fcedc0b0770 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcec80015c0 con 0x7fcedc0b0770 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 10 monclient(hunting): dump: 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.736 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 716627092 0 0) 0x7fcec0004eb0 con 0x7fcedc0b0770 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.749+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 >> v1:172.21.15.155:6790/0 conn(0x7fcedc14a700 legacy=0x7fcedc1dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 >> v1:172.21.15.155:6789/0 conn(0x7fcedc0ac5a0 legacy=0x7fcedc144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 1 monclient: found mon.b 2024-03-29T10:55:51.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc145990 con 0x7fcedc0b0770 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 10 monclient: _finish_auth 0 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fcee2da3640 5 monclient: authenticate success, global_id 4357 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fcee2da3640 10 monclient: _renew_subs 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec8001ad0 con 0x7fcedc0b0770 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fcee2da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcedc145320 con 0x7fcedc0b0770 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fcee2da3640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:51.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.750+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcec8001d80 con 0x7fcedc0b0770 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fcedaffd640 4 set_mon_vals no callback set 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fcee2da3640 10 monclient: _renew_subs 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fcee2da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcedc071400 con 0x7fcedc0b0770 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fcee2da3640 1 librados: init done 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fced97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fced97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.751+0000 7fced97fa640 10 monclient: dump: 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:51.739 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:51.740 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:51.740 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.754+0000 7fcee2da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.754+0000 7fcee2da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.754+0000 7fcee2da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.754+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc1df610 con 0x7fcedc0b0770 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.755+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fcec8019600 con 0x7fcedc0b0770 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.756+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fcec80835a0 con 0x7fcedc0b0770 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.756+0000 7fced97fa640 10 client.4357.objecter ms_dispatch 0x7fcedc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.756+0000 7fcee0b3f640 10 monclient: get_auth_request con 0x7fcec006df90 auth_method 0 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.757+0000 7fced97fa640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.757+0000 7fced97fa640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.757+0000 7fced97fa640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T10:55:51.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.757+0000 7fced97fa640 10 client.4357.objecter ms_handle_connect 0x7fcec006df90 2024-03-29T10:55:51.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.765+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcec804ee80 con 0x7fcedc0b0770 2024-03-29T10:55:51.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.765+0000 7fced97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:51.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:51.765+0000 7fced97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:52.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.089+0000 7fcee2da3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:52.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.089+0000 7fcee2da3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:52.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.089+0000 7fcee2da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:52.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.089+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcedc075f10 con 0x7fcedc0b0770 2024-03-29T10:55:52.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.090+0000 7fced97fa640 1 -- 172.21.15.155:0/2394287606 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fcec8050db0 con 0x7fcedc0b0770 2024-03-29T10:55:52.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.090+0000 7fced97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:52.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.090+0000 7fced97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:52.071 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:52.071 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:52.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 10 librados: watch_flush enter 2024-03-29T10:55:52.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 10 librados: watch_flush exit 2024-03-29T10:55:52.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T10:55:52.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 10 client.4357.objecter successfully canceled tick 2024-03-29T10:55:52.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 >> v1:172.21.15.183:6816/1412921548 conn(0x7fcec006df90 legacy=0x7fcec0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 10 monclient: shutdown 2024-03-29T10:55:52.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:52.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.094+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 >> v1:172.21.15.183:6789/0 conn(0x7fcedc0b0770 legacy=0x7fcedc1db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:52.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.095+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 shutdown_connections 2024-03-29T10:55:52.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.095+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 >> 172.21.15.155:0/2394287606 conn(0x7fcedc06b660 msgr2=0x7fcedc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:52.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.096+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 shutdown_connections 2024-03-29T10:55:52.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.096+0000 7fcee2da3640 1 -- 172.21.15.155:0/2394287606 wait complete. 2024-03-29T10:55:52.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:52.096+0000 7fcee2da3640 1 librados: shutdown 2024-03-29T10:55:52.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:58.902 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:59.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.076+0000 7fbc490c1640 10 monclient: get_monmap_and_config 2024-03-29T10:55:59.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.076+0000 7fbc490c1640 10 monclient: build_initial_monmap 2024-03-29T10:55:59.058 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.076+0000 7fbc490c1640 10 monclient: monmap: 2024-03-29T10:55:59.058 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:59.058 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:59.058 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:55:59.076723+0000 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:55:59.076723+0000 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.079+0000 7fbc490c1640 1 Processor -- start 2024-03-29T10:55:59.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.080+0000 7fbc490c1640 1 -- start start 2024-03-29T10:55:59.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.080+0000 7fbc490c1640 10 monclient: init 2024-03-29T10:55:59.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.080+0000 7fbc490c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:59.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.080+0000 7fbc490c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:59.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.080+0000 7fbc490c1640 10 monclient(hunting): picked mon.noname-a con 0x7fbc440b4100 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:59.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.081+0000 7fbc490c1640 10 monclient(hunting): picked mon.noname-b con 0x7fbc4413ee90 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.082+0000 7fbc490c1640 10 monclient(hunting): picked mon.noname-c con 0x7fbc440af900 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.082+0000 7fbc490c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440afd10 con 0x7fbc440b4100 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.082+0000 7fbc490c1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440af6f0 con 0x7fbc440af900 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.082+0000 7fbc490c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440aaf60 con 0x7fbc4413ee90 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.082+0000 7fbc490c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.082+0000 7fbc490c1640 10 monclient(hunting): authenticate will time out at 1062.374265s 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.083+0000 7fbc4359e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbc440af900 0x7fbc4413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:34907/0 (socket says 172.21.15.155:34907) 2024-03-29T10:55:59.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.083+0000 7fbc4359e640 1 -- 172.21.15.155:0/668102604 learned_addr learned my addr 172.21.15.155:0/668102604 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120516812 0 0) 0x7fbc440af6f0 con 0x7fbc440af900 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c003520 con 0x7fbc440af900 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554153512 0 0) 0x7fbc2c003520 con 0x7fbc440af900 2024-03-29T10:55:59.065 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.084+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 >> v1:172.21.15.155:6790/0 conn(0x7fbc4413ee90 legacy=0x7fbc440adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:59.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 >> v1:172.21.15.155:6789/0 conn(0x7fbc440b4100 legacy=0x7fbc440a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 1 monclient: found mon.noname-c 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4413e970 con 0x7fbc440af900 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: _finish_auth 0 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbc38001690 con 0x7fbc440af900 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: dump: 2024-03-29T10:55:59.069 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc490c1640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:55:59.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc490c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:59.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc38001860 con 0x7fbc440af900 2024-03-29T10:55:59.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:59.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 1 -- 172.21.15.155:0/668102604 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbc38001c70 con 0x7fbc440af900 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc41d9b640 10 monclient: dump: 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc490c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:59.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.088+0000 7fbc490c1640 4 set_mon_vals no callback set 2024-03-29T10:55:59.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.089+0000 7fbc490c1640 10 monclient: shutdown 2024-03-29T10:55:59.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.089+0000 7fbc490c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:59.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.089+0000 7fbc490c1640 1 -- 172.21.15.155:0/668102604 >> v1:172.21.15.183:6789/0 conn(0x7fbc440af900 legacy=0x7fbc4413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.091+0000 7fbc490c1640 1 -- 172.21.15.155:0/668102604 shutdown_connections 2024-03-29T10:55:59.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.091+0000 7fbc490c1640 1 -- 172.21.15.155:0/668102604 >> 172.21.15.155:0/668102604 conn(0x7fbc440b35e0 msgr2=0x7fbc440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:59.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.091+0000 7fbc490c1640 1 -- 172.21.15.155:0/668102604 shutdown_connections 2024-03-29T10:55:59.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.091+0000 7fbc490c1640 1 -- 172.21.15.155:0/668102604 wait complete. 2024-03-29T10:55:59.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.092+0000 7fbc490c1640 10 monclient: build_initial_monmap 2024-03-29T10:55:59.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.093+0000 7fbc490c1640 10 monclient: monmap: 2024-03-29T10:55:59.075 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:55:59.075 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.093+0000 7fbc490c1640 1 librados: starting msgr at 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.093+0000 7fbc490c1640 1 librados: starting objecter 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.094+0000 7fbc490c1640 1 Processor -- start 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.094+0000 7fbc490c1640 1 -- start start 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.094+0000 7fbc490c1640 1 librados: setting wanted keys 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.094+0000 7fbc490c1640 1 librados: calling monclient init 2024-03-29T10:55:59.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.094+0000 7fbc490c1640 10 monclient: init 2024-03-29T10:55:59.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.094+0000 7fbc490c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:59.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.095+0000 7fbc490c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:59.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.095+0000 7fbc490c1640 10 monclient(hunting): picked mon.noname-a con 0x7fbc440af900 addr v1:172.21.15.155:6789/0 2024-03-29T10:55:59.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.095+0000 7fbc490c1640 10 monclient(hunting): picked mon.noname-b con 0x7fbc440b4100 addr v1:172.21.15.183:6789/0 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc490c1640 10 monclient(hunting): picked mon.noname-c con 0x7fbc4413ee90 addr v1:172.21.15.155:6790/0 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc490c1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc441d9360 con 0x7fbc440af900 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc490c1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440703e0 con 0x7fbc440b4100 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc490c1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440706c0 con 0x7fbc4413ee90 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc490c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc490c1640 10 monclient(hunting): authenticate will time out at 1062.391259s 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc4259c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbc440b4100 0x7fbc441d8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:35001/0 (socket says 172.21.15.155:35001) 2024-03-29T10:55:59.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.099+0000 7fbc4259c640 1 -- 172.21.15.155:0/1190377678 learned_addr learned my addr 172.21.15.155:0/1190377678 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:55:59.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.100+0000 7fbc2b7fe640 10 client.?.objecter ms_handle_connect 0x7fbc440b4100 2024-03-29T10:55:59.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.100+0000 7fbc2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 client.?.objecter ms_handle_connect 0x7fbc440af900 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78699106 0 0) 0x7fbc441d9360 con 0x7fbc440af900 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:55:59.084 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc30004eb0 con 0x7fbc440af900 2024-03-29T10:55:59.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 client.?.objecter ms_handle_connect 0x7fbc4413ee90 2024-03-29T10:55:59.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.103+0000 7fbc2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:59.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.104+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006133852 0 0) 0x7fbc440703e0 con 0x7fbc440b4100 2024-03-29T10:55:59.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.104+0000 7fbc2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.085 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.104+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.104+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.104+0000 7fbc2b7fe640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.104+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc441d9360 con 0x7fbc440b4100 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2844128292 0 0) 0x7fbc441d9360 con 0x7fbc440b4100 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc30005490 con 0x7fbc440b4100 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbc34001710 con 0x7fbc440b4100 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:59.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471176720 0 0) 0x7fbc440706c0 con 0x7fbc4413ee90 2024-03-29T10:55:59.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:59.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:59.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:59.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:55:59.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc441d9360 con 0x7fbc4413ee90 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 568260800 0 0) 0x7fbc30004eb0 con 0x7fbc440af900 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc440706c0 con 0x7fbc440af900 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707055824 0 0) 0x7fbc441d9360 con 0x7fbc4413ee90 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc300082f0 con 0x7fbc4413ee90 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbc380019d0 con 0x7fbc4413ee90 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.092 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbc3c001c30 con 0x7fbc440af900 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.093 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1884666800 0 0) 0x7fbc30005490 con 0x7fbc440b4100 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 >> v1:172.21.15.155:6790/0 conn(0x7fbc4413ee90 legacy=0x7fbc441d8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 >> v1:172.21.15.155:6789/0 conn(0x7fbc440af900 legacy=0x7fbc4406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 monclient: found mon.b 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:59.094 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:59.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440709a0 con 0x7fbc440b4100 2024-03-29T10:55:59.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:59.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:59.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:59.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.109+0000 7fbc490c1640 5 monclient: authenticate success, global_id 4363 2024-03-29T10:55:59.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 10 monclient: _renew_subs 2024-03-29T10:55:59.095 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc34001c20 con 0x7fbc440b4100 2024-03-29T10:55:59.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:59.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4413de80 con 0x7fbc440b4100 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 10 monclient: _renew_subs 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4406f960 con 0x7fbc440b4100 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc490c1640 1 librados: init done 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.110+0000 7fbc2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.111+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbc34004a00 con 0x7fbc440b4100 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.111+0000 7fbc2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.111+0000 7fbc2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.111+0000 7fbc2b7fe640 10 monclient: dump: 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:55:59.098 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.111+0000 7fbc40d99640 4 set_mon_vals no callback set 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.113+0000 7fbc490c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.113+0000 7fbc490c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.113+0000 7fbc490c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.113+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4406dfd0 con 0x7fbc440b4100 2024-03-29T10:55:59.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.115+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fbc3401a9d0 con 0x7fbc440b4100 2024-03-29T10:55:59.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.116+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fbc340848a0 con 0x7fbc440b4100 2024-03-29T10:55:59.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.116+0000 7fbc2b7fe640 10 client.4363.objecter ms_dispatch 0x7fbc44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:59.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.116+0000 7fbc2b7fe640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:59.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.116+0000 7fbc2b7fe640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:59.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.116+0000 7fbc2b7fe640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T10:55:59.100 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.117+0000 7fbc42d9d640 10 monclient: get_auth_request con 0x7fbc3006e5a0 auth_method 0 2024-03-29T10:55:59.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.120+0000 7fbc2b7fe640 10 client.4363.objecter ms_handle_connect 0x7fbc3006e5a0 2024-03-29T10:55:59.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.121+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc34050180 con 0x7fbc440b4100 2024-03-29T10:55:59.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.121+0000 7fbc2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:59.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.121+0000 7fbc2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:59.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.448+0000 7fbc490c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:59.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.448+0000 7fbc490c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:59.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.448+0000 7fbc490c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:55:59.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.448+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc44075f10 con 0x7fbc440b4100 2024-03-29T10:55:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.453+0000 7fbc2b7fe640 1 -- 172.21.15.155:0/1190377678 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fbc340520b0 con 0x7fbc440b4100 2024-03-29T10:55:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.453+0000 7fbc2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.453+0000 7fbc2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:59.433 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:55:59.434 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.457+0000 7fbc490c1640 10 librados: watch_flush enter 2024-03-29T10:55:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.457+0000 7fbc490c1640 10 librados: watch_flush exit 2024-03-29T10:55:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.457+0000 7fbc490c1640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T10:55:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.457+0000 7fbc490c1640 10 client.4363.objecter successfully canceled tick 2024-03-29T10:55:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.458+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 >> v1:172.21.15.183:6816/1412921548 conn(0x7fbc3006e5a0 legacy=0x7fbc30070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.458+0000 7fbc490c1640 10 monclient: shutdown 2024-03-29T10:55:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.458+0000 7fbc490c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.458+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 >> v1:172.21.15.183:6789/0 conn(0x7fbc440b4100 legacy=0x7fbc441d8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.458+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 shutdown_connections 2024-03-29T10:55:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.458+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 >> 172.21.15.155:0/1190377678 conn(0x7fbc440b35e0 msgr2=0x7fbc440ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.459+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 shutdown_connections 2024-03-29T10:55:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.459+0000 7fbc490c1640 1 -- 172.21.15.155:0/1190377678 wait complete. 2024-03-29T10:55:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:55:59.460+0000 7fbc490c1640 1 librados: shutdown 2024-03-29T10:55:59.460 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:06.266 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:06.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.449+0000 7fdd786e2640 10 monclient: get_monmap_and_config 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.449+0000 7fdd786e2640 10 monclient: build_initial_monmap 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.450+0000 7fdd786e2640 10 monclient: monmap: 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:06.450645+0000 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:06.450645+0000 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:06.431 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:06.432 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:06.432 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:06.432 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:06.432 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:06.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.451+0000 7fdd786e2640 1 Processor -- start 2024-03-29T10:56:06.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.451+0000 7fdd786e2640 1 -- start start 2024-03-29T10:56:06.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.451+0000 7fdd786e2640 10 monclient: init 2024-03-29T10:56:06.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:06.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 10 monclient(hunting): picked mon.noname-a con 0x7fdd7006d9d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 10 monclient(hunting): picked mon.noname-b con 0x7fdd7013ef30 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdd700b07d0 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd700aed10 con 0x7fdd7006d9d0 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd700b0c40 con 0x7fdd700b07d0 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd700ab1b0 con 0x7fdd7013ef30 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:06.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.452+0000 7fdd786e2640 10 monclient(hunting): authenticate will time out at 1069.744470s 2024-03-29T10:56:06.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.461+0000 7fdd76c7f640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdd700b07d0 0x7fdd7014bd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:58621/0 (socket says 172.21.15.155:58621) 2024-03-29T10:56:06.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.461+0000 7fdd76c7f640 1 -- 172.21.15.155:0/56069049 learned_addr learned my addr 172.21.15.155:0/56069049 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.462+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396454056 0 0) 0x7fdd700b0c40 con 0x7fdd700b07d0 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.462+0000 7fdd7547c640 10 monclient(hunting): _negotiate 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.462+0000 7fdd7547c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.462+0000 7fdd7547c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.462+0000 7fdd7547c640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.462+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd60003520 con 0x7fdd700b07d0 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2891401470 0 0) 0x7fdd60003520 con 0x7fdd700b07d0 2024-03-29T10:56:06.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 >> v1:172.21.15.155:6790/0 conn(0x7fdd7013ef30 legacy=0x7fdd700af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 >> v1:172.21.15.155:6789/0 conn(0x7fdd7006d9d0 legacy=0x7fdd7013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: _finish_hunting 0 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 monclient: found mon.noname-c 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd700ab490 con 0x7fdd700b07d0 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: _finish_auth 0 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: _check_auth_tickets 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdd5c001690 con 0x7fdd700b07d0 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:06.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: dump: 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd786e2640 5 monclient: authenticate success, global_id 4366 2024-03-29T10:56:06.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd786e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd5c001860 con 0x7fdd700b07d0 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd786e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd786e2640 4 set_mon_vals no callback set 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 1 -- 172.21.15.155:0/56069049 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdd5c001cb0 con 0x7fdd700b07d0 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.463+0000 7fdd7547c640 10 monclient: dump: 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:06.447 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.464+0000 7fdd786e2640 10 monclient: shutdown 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.464+0000 7fdd786e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.464+0000 7fdd786e2640 1 -- 172.21.15.155:0/56069049 >> v1:172.21.15.183:6789/0 conn(0x7fdd700b07d0 legacy=0x7fdd7014bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.465+0000 7fdd786e2640 1 -- 172.21.15.155:0/56069049 shutdown_connections 2024-03-29T10:56:06.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.465+0000 7fdd786e2640 1 -- 172.21.15.155:0/56069049 >> 172.21.15.155:0/56069049 conn(0x7fdd7006b660 msgr2=0x7fdd7006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:06.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.466+0000 7fdd786e2640 1 -- 172.21.15.155:0/56069049 shutdown_connections 2024-03-29T10:56:06.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.469+0000 7fdd786e2640 1 -- 172.21.15.155:0/56069049 wait complete. 2024-03-29T10:56:06.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 10 monclient: build_initial_monmap 2024-03-29T10:56:06.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 10 monclient: monmap: 2024-03-29T10:56:06.454 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:06.454 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:06.454 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 1 librados: starting msgr at 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 1 librados: starting objecter 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 1 Processor -- start 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 1 -- start start 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 1 librados: setting wanted keys 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 1 librados: calling monclient init 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.470+0000 7fdd786e2640 10 monclient: init 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:06.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdd7006d9d0 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 10 monclient(hunting): picked mon.noname-b con 0x7fdd700b07d0 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 10 monclient(hunting): picked mon.noname-a con 0x7fdd7006f170 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd70070620 con 0x7fdd7006f170 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd70070900 con 0x7fdd700b07d0 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd701e79f0 con 0x7fdd7006d9d0 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:06.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.471+0000 7fdd786e2640 10 monclient(hunting): authenticate will time out at 1069.763541s 2024-03-29T10:56:06.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd75c7d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdd700b07d0 0x7fdd701229e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:58721/0 (socket says 172.21.15.155:58721) 2024-03-29T10:56:06.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd75c7d640 1 -- 172.21.15.155:0/2842416791 learned_addr learned my addr 172.21.15.155:0/2842416791 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:06.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd6effd640 10 client.?.objecter ms_handle_connect 0x7fdd700b07d0 2024-03-29T10:56:06.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:06.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283853653 0 0) 0x7fdd70070900 con 0x7fdd700b07d0 2024-03-29T10:56:06.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd6effd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:06.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:06.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.476+0000 7fdd6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:06.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.477+0000 7fdd6effd640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:56:06.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.477+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54004eb0 con 0x7fdd700b07d0 2024-03-29T10:56:06.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.478+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 507171689 0 0) 0x7fdd54004eb0 con 0x7fdd700b07d0 2024-03-29T10:56:06.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.478+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd70070900 con 0x7fdd700b07d0 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.478+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdd64001c30 con 0x7fdd700b07d0 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.478+0000 7fdd6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.478+0000 7fdd6effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.478+0000 7fdd6effd640 10 monclient(hunting): dump: 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.460 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:06.461 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:06.461 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:06.461 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:06.461 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:06.461 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:06.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.478+0000 7fdd6effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:06.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 client.?.objecter ms_handle_connect 0x7fdd7006f170 2024-03-29T10:56:06.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:06.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 client.?.objecter ms_handle_connect 0x7fdd7006d9d0 2024-03-29T10:56:06.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 600682180 0 0) 0x7fdd70070900 con 0x7fdd700b07d0 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 >> v1:172.21.15.155:6790/0 conn(0x7fdd7006d9d0 legacy=0x7fdd7006e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 >> v1:172.21.15.155:6789/0 conn(0x7fdd7006f170 legacy=0x7fdd70129be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 1 monclient: found mon.b 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd701230f0 con 0x7fdd700b07d0 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 monclient: _finish_auth 0 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:06.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:56:06.467 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 10 monclient: _renew_subs 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd70123340 con 0x7fdd700b07d0 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 10 monclient: _renew_subs 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7012b4e0 con 0x7fdd700b07d0 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.483+0000 7fdd786e2640 1 librados: init done 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.484+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd64003970 con 0x7fdd700b07d0 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.484+0000 7fdd6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.484+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdd64003dc0 con 0x7fdd700b07d0 2024-03-29T10:56:06.470 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.484+0000 7fdd6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.484+0000 7fdd6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.484+0000 7fdd6effd640 10 monclient: dump: 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.486+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fdd64003f20 con 0x7fdd700b07d0 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.487+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fdd64084e00 con 0x7fdd700b07d0 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.487+0000 7fdd6effd640 10 client.4369.objecter ms_dispatch 0x7fdd70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:06.471 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.487+0000 7fdd6effd640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.487+0000 7fdd6effd640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.487+0000 7fdd6effd640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.487+0000 7fdd7647e640 10 monclient: get_auth_request con 0x7fdd5406dc70 auth_method 0 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.488+0000 7fdd6ffff640 4 set_mon_vals no callback set 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.488+0000 7fdd6effd640 10 client.4369.objecter ms_handle_connect 0x7fdd5406dc70 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.489+0000 7fdd786e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.489+0000 7fdd786e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.489+0000 7fdd786e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:06.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.489+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd58005080 con 0x7fdd700b07d0 2024-03-29T10:56:06.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.495+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd640506e0 con 0x7fdd700b07d0 2024-03-29T10:56:06.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.495+0000 7fdd6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:06.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.495+0000 7fdd6effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:06.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.848+0000 7fdd786e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:06.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.848+0000 7fdd786e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:06.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.848+0000 7fdd786e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:06.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.848+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd58005330 con 0x7fdd700b07d0 2024-03-29T10:56:06.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.852+0000 7fdd6effd640 1 -- 172.21.15.155:0/2842416791 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fdd64052610 con 0x7fdd700b07d0 2024-03-29T10:56:06.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.852+0000 7fdd6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:06.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.852+0000 7fdd6effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:06.833 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:06.833 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:06.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.856+0000 7fdd786e2640 10 librados: watch_flush enter 2024-03-29T10:56:06.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.856+0000 7fdd786e2640 10 librados: watch_flush exit 2024-03-29T10:56:06.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.856+0000 7fdd786e2640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T10:56:06.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.856+0000 7fdd786e2640 10 client.4369.objecter successfully canceled tick 2024-03-29T10:56:06.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.859+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 >> v1:172.21.15.183:6816/1412921548 conn(0x7fdd5406dc70 legacy=0x7fdd54070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:06.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.859+0000 7fdd786e2640 10 monclient: shutdown 2024-03-29T10:56:06.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.859+0000 7fdd786e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:06.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.859+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 >> v1:172.21.15.183:6789/0 conn(0x7fdd700b07d0 legacy=0x7fdd701229e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:06.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.859+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 shutdown_connections 2024-03-29T10:56:06.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.859+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 >> 172.21.15.155:0/2842416791 conn(0x7fdd7006b660 msgr2=0x7fdd7013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:06.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.862+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 shutdown_connections 2024-03-29T10:56:06.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.863+0000 7fdd786e2640 1 -- 172.21.15.155:0/2842416791 wait complete. 2024-03-29T10:56:06.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:06.863+0000 7fdd786e2640 1 librados: shutdown 2024-03-29T10:56:06.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:13.673 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:13.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.843+0000 7f574596b640 10 monclient: get_monmap_and_config 2024-03-29T10:56:13.824 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.843+0000 7f574596b640 10 monclient: build_initial_monmap 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.843+0000 7f574596b640 10 monclient: monmap: 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:13.843948+0000 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:13.843948+0000 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:13.825 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:13.826 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.844+0000 7f574596b640 1 Processor -- start 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.844+0000 7f574596b640 1 -- start start 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.844+0000 7f574596b640 10 monclient: init 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.845+0000 7f574596b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.845+0000 7f574596b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.845+0000 7f574596b640 10 monclient(hunting): picked mon.noname-b con 0x7f57400a9750 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.846+0000 7f574596b640 10 monclient(hunting): picked mon.noname-c con 0x7f57400ae220 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.846+0000 7f5744969640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f57400a9750 0x7f5740141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57296/0 (socket says 172.21.15.155:57296) 2024-03-29T10:56:13.827 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.846+0000 7f5744969640 1 -- 172.21.15.155:0/3771925265 learned_addr learned my addr 172.21.15.155:0/3771925265 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:13.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.846+0000 7f574596b640 10 monclient(hunting): picked mon.noname-a con 0x7f574013b9d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:13.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.847+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5740142490 con 0x7f574013b9d0 2024-03-29T10:56:13.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.847+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400ac130 con 0x7f57400ae220 2024-03-29T10:56:13.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.847+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400ac410 con 0x7f57400a9750 2024-03-29T10:56:13.828 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.847+0000 7f574596b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:13.829 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.848+0000 7f574596b640 10 monclient(hunting): authenticate will time out at 1077.139881s 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624716122 0 0) 0x7f57400ac410 con 0x7f57400a9750 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728003520 con 0x7f57400a9750 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388960929 0 0) 0x7f5728003520 con 0x7f57400a9750 2024-03-29T10:56:13.831 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 >> v1:172.21.15.183:6789/0 conn(0x7f57400ae220 legacy=0x7f574013b2c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 >> v1:172.21.15.155:6789/0 conn(0x7f574013b9d0 legacy=0x7f5740149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.849+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400afa40 con 0x7f57400a9750 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f573f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f573f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f573f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5738001a10 con 0x7f57400a9750 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f573f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f573f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f573f7fe640 10 monclient: dump: 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:13.832 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f574596b640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:56:13.833 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.850+0000 7f574596b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.852+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5738001c30 con 0x7f57400a9750 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.852+0000 7f573f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.852+0000 7f573f7fe640 1 -- 172.21.15.155:0/3771925265 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5738001fc0 con 0x7f57400a9750 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.852+0000 7f573f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.852+0000 7f573f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.852+0000 7f573f7fe640 10 monclient: dump: 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:13.835 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.853+0000 7f574596b640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.853+0000 7f574596b640 4 set_mon_vals no callback set 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.853+0000 7f574596b640 10 monclient: shutdown 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.854+0000 7f574596b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.854+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 >> v1:172.21.15.155:6790/0 conn(0x7f57400a9750 legacy=0x7f5740141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:13.836 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.855+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 shutdown_connections 2024-03-29T10:56:13.837 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.855+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 >> 172.21.15.155:0/3771925265 conn(0x7f57400673e0 msgr2=0x7f57400b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:13.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.857+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 shutdown_connections 2024-03-29T10:56:13.838 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.857+0000 7f574596b640 1 -- 172.21.15.155:0/3771925265 wait complete. 2024-03-29T10:56:13.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.858+0000 7f574596b640 10 monclient: build_initial_monmap 2024-03-29T10:56:13.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.858+0000 7f574596b640 10 monclient: monmap: 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.859+0000 7f574596b640 1 librados: starting msgr at 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.859+0000 7f574596b640 1 librados: starting objecter 2024-03-29T10:56:13.841 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.859+0000 7f574596b640 1 Processor -- start 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.859+0000 7f574596b640 1 -- start start 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.859+0000 7f574596b640 1 librados: setting wanted keys 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.859+0000 7f574596b640 1 librados: calling monclient init 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.859+0000 7f574596b640 10 monclient: init 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.860+0000 7f574596b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.860+0000 7f574596b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.860+0000 7f574596b640 10 monclient(hunting): picked mon.noname-c con 0x7f57400a9750 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f574596b640 10 monclient(hunting): picked mon.noname-a con 0x7f57400ae220 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f574596b640 10 monclient(hunting): picked mon.noname-b con 0x7f574013b9d0 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f574596b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5740072450 con 0x7f57400ae220 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f574596b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574006ffa0 con 0x7f574013b9d0 2024-03-29T10:56:13.843 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f574596b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574013ca20 con 0x7f57400a9750 2024-03-29T10:56:13.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f574596b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:13.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f574596b640 10 monclient(hunting): authenticate will time out at 1077.154669s 2024-03-29T10:56:13.844 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f5744969640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f57400a9750 0x7f57400719d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:59512/0 (socket says 172.21.15.155:59512) 2024-03-29T10:56:13.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.862+0000 7f5744969640 1 -- 172.21.15.155:0/2289702763 learned_addr learned my addr 172.21.15.155:0/2289702763 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.864+0000 7f573d7fa640 10 client.?.objecter ms_handle_connect 0x7f57400ae220 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746762356 0 0) 0x7f5740072450 con 0x7f57400ae220 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572c004eb0 con 0x7f57400ae220 2024-03-29T10:56:13.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 10 client.?.objecter ms_handle_connect 0x7f57400a9750 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3849008763 0 0) 0x7f572c004eb0 con 0x7f57400ae220 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5740072450 con 0x7f57400ae220 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.865+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5730001870 con 0x7f57400ae220 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:13.851 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686927633 0 0) 0x7f574013ca20 con 0x7f57400a9750 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572c004eb0 con 0x7f57400a9750 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2794000653 0 0) 0x7f5740072450 con 0x7f57400ae220 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 >> v1:172.21.15.155:6790/0 conn(0x7f57400a9750 legacy=0x7f57400719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:13.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 >> v1:172.21.15.183:6789/0 conn(0x7f574013b9d0 legacy=0x7f574006f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 1 monclient: found mon.a 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574013cd00 con 0x7f57400ae220 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f573d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f574596b640 5 monclient: authenticate success, global_id 4377 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f574596b640 10 monclient: _renew_subs 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f574596b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.866+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57400720e0 con 0x7f57400ae220 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f574596b640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f574596b640 10 monclient: _renew_subs 2024-03-29T10:56:13.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f574596b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f574014fa20 con 0x7f57400ae220 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f574596b640 1 librados: init done 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5730001e00 con 0x7f57400ae220 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f573d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5730003830 con 0x7f57400ae220 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f573d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f573d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.867+0000 7f573d7fa640 10 monclient: dump: 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:13.854 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:13.855 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:13.855 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:13.855 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:13.855 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:13.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.868+0000 7f573effd640 4 set_mon_vals no callback set 2024-03-29T10:56:13.855 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.869+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f5730019800 con 0x7f57400ae220 2024-03-29T10:56:13.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f5730083860 con 0x7f57400ae220 2024-03-29T10:56:13.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f573d7fa640 10 client.4377.objecter ms_dispatch 0x7f5740000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:13.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f573d7fa640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:13.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f573d7fa640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:13.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f573d7fa640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T10:56:13.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f5744969640 10 monclient: get_auth_request con 0x7f572c072160 auth_method 0 2024-03-29T10:56:13.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f574596b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:13.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f574596b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:13.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f574596b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:56:13.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5720005080 con 0x7f57400ae220 2024-03-29T10:56:13.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.870+0000 7f573d7fa640 10 client.4377.objecter ms_handle_connect 0x7f572c072160 2024-03-29T10:56:13.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.879+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f573004ef70 con 0x7f57400ae220 2024-03-29T10:56:13.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.879+0000 7f573d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:13.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:13.879+0000 7f573d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:14.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.206+0000 7f574596b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:14.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.206+0000 7f574596b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:14.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.206+0000 7f574596b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:56:14.188 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.206+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5720005330 con 0x7f57400ae220 2024-03-29T10:56:14.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.212+0000 7f573d7fa640 1 -- 172.21.15.155:0/2289702763 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f5730051070 con 0x7f57400ae220 2024-03-29T10:56:14.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.212+0000 7f573d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:14.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.212+0000 7f573d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:14.193 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:14.193 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:14.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.215+0000 7f574596b640 10 librados: watch_flush enter 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 10 librados: watch_flush exit 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 10 client.4377.objecter successfully canceled tick 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 >> v1:172.21.15.183:6816/1412921548 conn(0x7f572c072160 legacy=0x7f572c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 10 monclient: shutdown 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 >> v1:172.21.15.155:6789/0 conn(0x7f57400ae220 legacy=0x7f574006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 shutdown_connections 2024-03-29T10:56:14.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.216+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 >> 172.21.15.155:0/2289702763 conn(0x7f57400673e0 msgr2=0x7f57400b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:14.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.217+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 shutdown_connections 2024-03-29T10:56:14.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.217+0000 7f574596b640 1 -- 172.21.15.155:0/2289702763 wait complete. 2024-03-29T10:56:14.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:14.217+0000 7f574596b640 1 librados: shutdown 2024-03-29T10:56:14.214 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:21.022 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:21.181 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.200+0000 7fb6227db640 10 monclient: get_monmap_and_config 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.200+0000 7fb6227db640 10 monclient: build_initial_monmap 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.200+0000 7fb6227db640 10 monclient: monmap: 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:21.202145+0000 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:21.202145+0000 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:21.182 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:21.183 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.202+0000 7fb6227db640 1 Processor -- start 2024-03-29T10:56:21.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.203+0000 7fb6227db640 1 -- start start 2024-03-29T10:56:21.184 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.203+0000 7fb6227db640 10 monclient: init 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c1457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c13f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c0a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c0aaef0 con 0x7fb61c1457f0 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c145bd0 con 0x7fb61c0a9d70 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb61bfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb61c1457f0 0x7fb61c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:47208/0 (socket says 172.21.15.155:47208) 2024-03-29T10:56:21.186 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb61bfff640 1 -- 172.21.15.155:0/1897546473 learned_addr learned my addr 172.21.15.155:0/1897546473 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:21.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 1 -- 172.21.15.155:0/1897546473 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c145eb0 con 0x7fb61c13f650 2024-03-29T10:56:21.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:21.187 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.204+0000 7fb6227db640 10 monclient(hunting): authenticate will time out at 1084.497843s 2024-03-29T10:56:21.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.205+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362109357 0 0) 0x7fb61c145bd0 con 0x7fb61c0a9d70 2024-03-29T10:56:21.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.205+0000 7fb61affd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.205+0000 7fb61affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.205+0000 7fb61affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.205+0000 7fb61affd640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.205+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb604003520 con 0x7fb61c0a9d70 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981761938 0 0) 0x7fb61c0aaef0 con 0x7fb61c1457f0 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c145bd0 con 0x7fb61c1457f0 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021592773 0 0) 0x7fb61c145eb0 con 0x7fb61c13f650 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.192 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c0aaef0 con 0x7fb61c13f650 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3541980752 0 0) 0x7fb604003520 con 0x7fb61c0a9d70 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 >> v1:172.21.15.155:6790/0 conn(0x7fb61c13f650 legacy=0x7fb61c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 >> v1:172.21.15.155:6789/0 conn(0x7fb61c1457f0 legacy=0x7fb61c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 monclient: found mon.noname-c 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c146340 con 0x7fb61c0a9d70 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient: _finish_auth 0 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:21.193 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb610001690 con 0x7fb61c0a9d70 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb6227db640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb6227db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb61affd640 10 monclient: dump: 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:21.194 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.206+0000 7fb6227db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6100018b0 con 0x7fb61c0a9d70 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb61affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb61affd640 1 -- 172.21.15.155:0/1897546473 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb610001c40 con 0x7fb61c0a9d70 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb6227db640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb6227db640 4 set_mon_vals no callback set 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb61affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb61affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.207+0000 7fb61affd640 10 monclient: dump: 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.195 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.208+0000 7fb6227db640 10 monclient: shutdown 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.208+0000 7fb6227db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.208+0000 7fb6227db640 1 -- 172.21.15.155:0/1897546473 >> v1:172.21.15.183:6789/0 conn(0x7fb61c0a9d70 legacy=0x7fb61c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.208+0000 7fb6227db640 1 -- 172.21.15.155:0/1897546473 shutdown_connections 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.208+0000 7fb6227db640 1 -- 172.21.15.155:0/1897546473 >> 172.21.15.155:0/1897546473 conn(0x7fb61c066a00 msgr2=0x7fb61c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.208+0000 7fb6227db640 1 -- 172.21.15.155:0/1897546473 shutdown_connections 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.208+0000 7fb6227db640 1 -- 172.21.15.155:0/1897546473 wait complete. 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.209+0000 7fb6227db640 10 monclient: build_initial_monmap 2024-03-29T10:56:21.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.209+0000 7fb6227db640 10 monclient: monmap: 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.210+0000 7fb6227db640 1 librados: starting msgr at 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.210+0000 7fb6227db640 1 librados: starting objecter 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.210+0000 7fb6227db640 1 Processor -- start 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.210+0000 7fb6227db640 1 -- start start 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.210+0000 7fb6227db640 1 librados: setting wanted keys 2024-03-29T10:56:21.197 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.210+0000 7fb6227db640 1 librados: calling monclient init 2024-03-29T10:56:21.198 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.210+0000 7fb6227db640 10 monclient: init 2024-03-29T10:56:21.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.211+0000 7fb6227db640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:21.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.211+0000 7fb6227db640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:21.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c0a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:21.202 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c13f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c1457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb61b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fb61c13f650 0x7fb61c1cbcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:58516/0 (socket says 172.21.15.155:58516) 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb61b7fe640 1 -- 172.21.15.155:0/2611084310 learned_addr learned my addr 172.21.15.155:0/2611084310 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c1cef70 con 0x7fb61c1457f0 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c1cb790 con 0x7fb61c0a9d70 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c1cb340 con 0x7fb61c13f650 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 client.?.objecter ms_handle_connect 0x7fb61c13f650 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 client.?.objecter ms_handle_connect 0x7fb61c0a9d70 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb6227db640 10 monclient(hunting): authenticate will time out at 1084.505469s 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795899519 0 0) 0x7fb61c1cb790 con 0x7fb61c0a9d70 2024-03-29T10:56:21.203 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb608004eb0 con 0x7fb61c0a9d70 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 client.?.objecter ms_handle_connect 0x7fb61c1457f0 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.212+0000 7fb618ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4038828143 0 0) 0x7fb608004eb0 con 0x7fb61c0a9d70 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb61c1cb790 con 0x7fb61c0a9d70 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb60c001c90 con 0x7fb61c0a9d70 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:21.204 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594240419 0 0) 0x7fb61c1cef70 con 0x7fb61c1457f0 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:56:21.205 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb608004eb0 con 0x7fb61c1457f0 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3491223574 0 0) 0x7fb608004eb0 con 0x7fb61c1457f0 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb61c1cef70 con 0x7fb61c1457f0 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb610001430 con 0x7fb61c1457f0 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:21.206 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510073187 0 0) 0x7fb61c1cb340 con 0x7fb61c13f650 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb608004eb0 con 0x7fb61c13f650 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 7977276 0 0) 0x7fb61c1cb790 con 0x7fb61c0a9d70 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 >> v1:172.21.15.155:6790/0 conn(0x7fb61c13f650 legacy=0x7fb61c1cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 >> v1:172.21.15.155:6789/0 conn(0x7fb61c1457f0 legacy=0x7fb61c1e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 monclient: found mon.b 2024-03-29T10:56:21.207 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c1cb4d0 con 0x7fb61c0a9d70 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb618ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb6227db640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb6227db640 10 monclient: _renew_subs 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb6227db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.213+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c139ac0 con 0x7fb61c0a9d70 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb6227db640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb6227db640 10 monclient: _renew_subs 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb6227db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c142250 con 0x7fb61c0a9d70 2024-03-29T10:56:21.208 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb6227db640 1 librados: init done 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb60c004b60 con 0x7fb61c0a9d70 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb618ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb60c004e70 con 0x7fb61c0a9d70 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb618ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb618ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb618ff9640 10 monclient: dump: 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:21.209 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.214+0000 7fb61a7fc640 4 set_mon_vals no callback set 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.216+0000 7fb6227db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.216+0000 7fb6227db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.216+0000 7fb6227db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.216+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc005080 con 0x7fb61c0a9d70 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.217+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fb60c01ae40 con 0x7fb61c0a9d70 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.218+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fb60c084d30 con 0x7fb61c0a9d70 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.218+0000 7fb618ff9640 10 client.4375.objecter ms_dispatch 0x7fb61c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.218+0000 7fb618ff9640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.218+0000 7fb618ff9640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.218+0000 7fb618ff9640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.218+0000 7fb61b7fe640 10 monclient: get_auth_request con 0x7fb60806e0f0 auth_method 0 2024-03-29T10:56:21.210 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.218+0000 7fb618ff9640 10 client.4375.objecter ms_handle_connect 0x7fb60806e0f0 2024-03-29T10:56:21.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.223+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb60c050650 con 0x7fb61c0a9d70 2024-03-29T10:56:21.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.223+0000 7fb618ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:21.211 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.223+0000 7fb618ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:21.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.621+0000 7fb6227db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:21.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.622+0000 7fb6227db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:21.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.622+0000 7fb6227db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:21.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.622+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5fc005330 con 0x7fb61c0a9d70 2024-03-29T10:56:21.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.623+0000 7fb618ff9640 1 -- 172.21.15.155:0/2611084310 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fb60c052580 con 0x7fb61c0a9d70 2024-03-29T10:56:21.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.623+0000 7fb618ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:21.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.623+0000 7fb618ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:21.604 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:21.605 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 10 librados: watch_flush enter 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 10 librados: watch_flush exit 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 10 client.4375.objecter successfully canceled tick 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 >> v1:172.21.15.183:6816/1412921548 conn(0x7fb60806e0f0 legacy=0x7fb6080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 10 monclient: shutdown 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 >> v1:172.21.15.183:6789/0 conn(0x7fb61c0a9d70 legacy=0x7fb61c1ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 shutdown_connections 2024-03-29T10:56:21.609 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.627+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 >> 172.21.15.155:0/2611084310 conn(0x7fb61c066a00 msgr2=0x7fb61c13ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:21.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.628+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 shutdown_connections 2024-03-29T10:56:21.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.628+0000 7fb6227db640 1 -- 172.21.15.155:0/2611084310 wait complete. 2024-03-29T10:56:21.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:21.628+0000 7fb6227db640 1 librados: shutdown 2024-03-29T10:56:21.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:28.432 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:28.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.591+0000 7fdffbc33640 10 monclient: get_monmap_and_config 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.591+0000 7fdffbc33640 10 monclient: build_initial_monmap 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.591+0000 7fdffbc33640 10 monclient: monmap: 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:28.592976+0000 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:28.592976+0000 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:28.573 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:28.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.593+0000 7fdffbc33640 1 Processor -- start 2024-03-29T10:56:28.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.593+0000 7fdffbc33640 1 -- start start 2024-03-29T10:56:28.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.593+0000 7fdffbc33640 10 monclient: init 2024-03-29T10:56:28.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 10 monclient(hunting): picked mon.noname-b con 0x7fdff40b0770 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 10 monclient(hunting): picked mon.noname-c con 0x7fdff40ad850 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdff99cf640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fdff40b0770 0x7fdff4072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42224/0 (socket says 172.21.15.155:42224) 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdff91ce640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdff40ad850 0x7fdff40aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:51882/0 (socket says 172.21.15.155:51882) 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 10 monclient(hunting): picked mon.noname-a con 0x7fdff413c1b0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdff99cf640 1 -- 172.21.15.155:0/4079102924 learned_addr learned my addr 172.21.15.155:0/4079102924 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff40ac8e0 con 0x7fdff413c1b0 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff40ad330 con 0x7fdff40ad850 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff40acfe0 con 0x7fdff40b0770 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.594+0000 7fdffbc33640 10 monclient(hunting): authenticate will time out at 1091.887841s 2024-03-29T10:56:28.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708007172 0 0) 0x7fdff40acfe0 con 0x7fdff40b0770 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe0003520 con 0x7fdff40b0770 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663040244 0 0) 0x7fdff40ad330 con 0x7fdff40ad850 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:56:28.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe0003840 con 0x7fdff40ad850 2024-03-29T10:56:28.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427954094 0 0) 0x7fdfe0003520 con 0x7fdff40b0770 2024-03-29T10:56:28.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 >> v1:172.21.15.183:6789/0 conn(0x7fdff40ad850 legacy=0x7fdff40aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:28.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.595+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 >> v1:172.21.15.155:6789/0 conn(0x7fdff413c1b0 legacy=0x7fdff414a760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 1 monclient: found mon.noname-b 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff40ab1f0 con 0x7fdff40b0770 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: _finish_auth 0 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdff0001670 con 0x7fdff40b0770 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:28.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: dump: 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdffbc33640 5 monclient: authenticate success, global_id 4409 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdffbc33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdff0001850 con 0x7fdff40b0770 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.597+0000 7fdff89cd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:28.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.598+0000 7fdff89cd640 1 -- 172.21.15.155:0/4079102924 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdff0001c60 con 0x7fdff40b0770 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.598+0000 7fdff89cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.598+0000 7fdff89cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.598+0000 7fdff89cd640 10 monclient: dump: 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.598+0000 7fdffbc33640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:28.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.598+0000 7fdffbc33640 4 set_mon_vals no callback set 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.599+0000 7fdffbc33640 10 monclient: shutdown 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.599+0000 7fdffbc33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.599+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 >> v1:172.21.15.155:6790/0 conn(0x7fdff40b0770 legacy=0x7fdff4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.600+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 shutdown_connections 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.600+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 >> 172.21.15.155:0/4079102924 conn(0x7fdff406b660 msgr2=0x7fdff406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.600+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 shutdown_connections 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.601+0000 7fdffbc33640 1 -- 172.21.15.155:0/4079102924 wait complete. 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.602+0000 7fdffbc33640 10 monclient: build_initial_monmap 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.602+0000 7fdffbc33640 10 monclient: monmap: 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:28.584 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.602+0000 7fdffbc33640 1 librados: starting msgr at 2024-03-29T10:56:28.585 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.603+0000 7fdffbc33640 1 librados: starting objecter 2024-03-29T10:56:28.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.603+0000 7fdffbc33640 1 Processor -- start 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.603+0000 7fdffbc33640 1 -- start start 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.603+0000 7fdffbc33640 1 librados: setting wanted keys 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.603+0000 7fdffbc33640 1 librados: calling monclient init 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.603+0000 7fdffbc33640 10 monclient: init 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 10 monclient(hunting): picked mon.noname-a con 0x7fdff40ad850 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 10 monclient(hunting): picked mon.noname-c con 0x7fdff40b0770 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 10 monclient(hunting): picked mon.noname-b con 0x7fdff413c1b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdff91ce640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fdff40b0770 0x7fdff4134830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:42770/0 (socket says 172.21.15.155:42770) 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdff91ce640 1 -- 172.21.15.155:0/4183025631 learned_addr learned my addr 172.21.15.155:0/4183025631 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff406e6b0 con 0x7fdff40ad850 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff4070400 con 0x7fdff413c1b0 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff406f160 con 0x7fdff40b0770 2024-03-29T10:56:28.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:28.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.605+0000 7fdffbc33640 10 monclient(hunting): authenticate will time out at 1091.898349s 2024-03-29T10:56:28.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 10 client.?.objecter ms_handle_connect 0x7fdff40b0770 2024-03-29T10:56:28.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:28.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472638070 0 0) 0x7fdff406f160 con 0x7fdff40b0770 2024-03-29T10:56:28.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:28.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.606+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd8004eb0 con 0x7fdff40b0770 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.607+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1443500189 0 0) 0x7fdfd8004eb0 con 0x7fdff40b0770 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.607+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdff406f160 con 0x7fdff40b0770 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.607+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdfe40019d0 con 0x7fdff40b0770 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient(hunting): dump: 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:28.591 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1207106448 0 0) 0x7fdff406f160 con 0x7fdff40b0770 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 >> v1:172.21.15.183:6789/0 conn(0x7fdff413c1b0 legacy=0x7fdff41d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 >> v1:172.21.15.155:6789/0 conn(0x7fdff40ad850 legacy=0x7fdff406dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 1 monclient: found mon.c 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff41d06d0 con 0x7fdff40b0770 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:28.592 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 client.?.objecter ms_handle_connect 0x7fdff413c1b0 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdfea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 5 monclient: authenticate success, global_id 4412 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 10 monclient: _renew_subs 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff406edc0 con 0x7fdff40b0770 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 10 monclient: _renew_subs 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff4135120 con 0x7fdff40b0770 2024-03-29T10:56:28.593 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.608+0000 7fdffbc33640 1 librados: init done 2024-03-29T10:56:28.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.609+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfe4001b30 con 0x7fdff40b0770 2024-03-29T10:56:28.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdfea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:28.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdffbc33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fdfe4004c30 con 0x7fdff40b0770 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdffbc33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdffbc33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff4070270 con 0x7fdff40b0770 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdfebfff640 4 set_mon_vals no callback set 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdfea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdfea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.610+0000 7fdfea7fc640 10 monclient: dump: 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:28.596 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.611+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fdfe401ac40 con 0x7fdff40b0770 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.612+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fdfe4084bf0 con 0x7fdff40b0770 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.612+0000 7fdff99cf640 10 monclient: get_auth_request con 0x7fdfd806dcd0 auth_method 0 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.612+0000 7fdfea7fc640 10 client.4412.objecter ms_dispatch 0x7fdff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.612+0000 7fdfea7fc640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.612+0000 7fdfea7fc640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.612+0000 7fdfea7fc640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T10:56:28.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.612+0000 7fdfea7fc640 10 client.4412.objecter ms_handle_connect 0x7fdfd806dcd0 2024-03-29T10:56:28.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.619+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfe40504d0 con 0x7fdff40b0770 2024-03-29T10:56:28.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.619+0000 7fdfea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:28.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.620+0000 7fdfea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:28.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.929+0000 7fdffbc33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:28.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.929+0000 7fdffbc33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:28.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.929+0000 7fdffbc33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:28.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.929+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdff406f2f0 con 0x7fdff40b0770 2024-03-29T10:56:28.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.930+0000 7fdfea7fc640 1 -- 172.21.15.155:0/4183025631 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fdfe4052400 con 0x7fdff40b0770 2024-03-29T10:56:28.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.930+0000 7fdfea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:28.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.930+0000 7fdfea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:28.912 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:28.912 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.933+0000 7fdffbc33640 10 librados: watch_flush enter 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.933+0000 7fdffbc33640 10 librados: watch_flush exit 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.933+0000 7fdffbc33640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 10 client.4412.objecter successfully canceled tick 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 >> v1:172.21.15.183:6816/1412921548 conn(0x7fdfd806dcd0 legacy=0x7fdfd8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 10 monclient: shutdown 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 >> v1:172.21.15.155:6790/0 conn(0x7fdff40b0770 legacy=0x7fdff4134830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 shutdown_connections 2024-03-29T10:56:28.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 >> 172.21.15.155:0/4183025631 conn(0x7fdff406b660 msgr2=0x7fdff40ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:28.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 shutdown_connections 2024-03-29T10:56:28.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 1 -- 172.21.15.155:0/4183025631 wait complete. 2024-03-29T10:56:28.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:28.934+0000 7fdffbc33640 1 librados: shutdown 2024-03-29T10:56:28.930 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:35.738 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:35.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.910+0000 7fc13d8e0640 10 monclient: get_monmap_and_config 2024-03-29T10:56:35.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.910+0000 7fc13d8e0640 10 monclient: build_initial_monmap 2024-03-29T10:56:35.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.910+0000 7fc13d8e0640 10 monclient: monmap: 2024-03-29T10:56:35.891 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:35.891 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:35.891 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:35.911201+0000 2024-03-29T10:56:35.892 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:35.911201+0000 2024-03-29T10:56:35.892 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:35.892 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:35.892 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:35.892 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:35.892 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:35.892 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.911+0000 7fc13d8e0640 1 Processor -- start 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.911+0000 7fc13d8e0640 1 -- start start 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.911+0000 7fc13d8e0640 10 monclient: init 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 10 monclient(hunting): picked mon.noname-c con 0x7fc1381422b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 10 monclient(hunting): picked mon.noname-b con 0x7fc138142690 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:35.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 10 monclient(hunting): picked mon.noname-a con 0x7fc1380b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc138142c80 con 0x7fc1380b17d0 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1380aae40 con 0x7fc1381422b0 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13813fbd0 con 0x7fc138142690 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc13d8e0640 10 monclient(hunting): authenticate will time out at 1099.204479s 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc1367fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc138142690 0x7fc1380b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43290/0 (socket says 172.21.15.155:43290) 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.912+0000 7fc1367fc640 1 -- 172.21.15.155:0/2209051579 learned_addr learned my addr 172.21.15.155:0/2209051579 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.913+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881674814 0 0) 0x7fc13813fbd0 con 0x7fc138142690 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.913+0000 7fc135ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.913+0000 7fc135ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:35.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.913+0000 7fc135ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:35.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.913+0000 7fc135ffb640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:56:35.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.913+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc120003520 con 0x7fc138142690 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1038488169 0 0) 0x7fc120003520 con 0x7fc138142690 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 >> v1:172.21.15.183:6789/0 conn(0x7fc1381422b0 legacy=0x7fc13813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 >> v1:172.21.15.155:6789/0 conn(0x7fc1380b17d0 legacy=0x7fc1380b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 1 monclient: found mon.noname-b 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:56:35.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138151940 con 0x7fc138142690 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc128001690 con 0x7fc138142690 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc135ffb640 10 monclient: dump: 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.898 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:35.899 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:35.899 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:35.899 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:35.899 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:35.899 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:35.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc13d8e0640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:56:35.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.916+0000 7fc13d8e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:35.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1280017f0 con 0x7fc138142690 2024-03-29T10:56:35.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc135ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:35.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc135ffb640 1 -- 172.21.15.155:0/2209051579 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc128001c40 con 0x7fc138142690 2024-03-29T10:56:35.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc135ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc135ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc135ffb640 10 monclient: dump: 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:35.901 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc13d8e0640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.917+0000 7fc13d8e0640 4 set_mon_vals no callback set 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.918+0000 7fc13d8e0640 10 monclient: shutdown 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.918+0000 7fc13d8e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.918+0000 7fc13d8e0640 1 -- 172.21.15.155:0/2209051579 >> v1:172.21.15.155:6790/0 conn(0x7fc138142690 legacy=0x7fc1380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.918+0000 7fc13d8e0640 1 -- 172.21.15.155:0/2209051579 shutdown_connections 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.918+0000 7fc13d8e0640 1 -- 172.21.15.155:0/2209051579 >> 172.21.15.155:0/2209051579 conn(0x7fc138066a00 msgr2=0x7fc138066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.919+0000 7fc13d8e0640 1 -- 172.21.15.155:0/2209051579 shutdown_connections 2024-03-29T10:56:35.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.919+0000 7fc13d8e0640 1 -- 172.21.15.155:0/2209051579 wait complete. 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.920+0000 7fc13d8e0640 10 monclient: build_initial_monmap 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.920+0000 7fc13d8e0640 10 monclient: monmap: 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:35.904 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.921+0000 7fc13d8e0640 1 librados: starting msgr at 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.921+0000 7fc13d8e0640 1 librados: starting objecter 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.921+0000 7fc13d8e0640 1 Processor -- start 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.921+0000 7fc13d8e0640 1 -- start start 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.921+0000 7fc13d8e0640 1 librados: setting wanted keys 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.921+0000 7fc13d8e0640 1 librados: calling monclient init 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.921+0000 7fc13d8e0640 10 monclient: init 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.922+0000 7fc13d8e0640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.922+0000 7fc13d8e0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:35.905 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.922+0000 7fc13d8e0640 10 monclient(hunting): picked mon.noname-c con 0x7fc1380b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:35.906 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.922+0000 7fc13d8e0640 10 monclient(hunting): picked mon.noname-a con 0x7fc1381422b0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc136ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc1380b17d0 0x7fc13806f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43594/0 (socket says 172.21.15.155:43594) 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc13d8e0640 10 monclient(hunting): picked mon.noname-b con 0x7fc138142690 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc13d8e0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1381509d0 con 0x7fc1381422b0 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc136ffd640 1 -- 172.21.15.155:0/3927588537 learned_addr learned my addr 172.21.15.155:0/3927588537 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13806e9f0 con 0x7fc138142690 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13806ecd0 con 0x7fc1380b17d0 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc13d8e0640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:35.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.925+0000 7fc13d8e0640 10 monclient(hunting): authenticate will time out at 1099.217729s 2024-03-29T10:56:35.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 client.?.objecter ms_handle_connect 0x7fc138142690 2024-03-29T10:56:35.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:35.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 client.?.objecter ms_handle_connect 0x7fc1380b17d0 2024-03-29T10:56:35.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:35.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625049703 0 0) 0x7fc13806e9f0 con 0x7fc138142690 2024-03-29T10:56:35.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc124004eb0 con 0x7fc138142690 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037279307 0 0) 0x7fc13806ecd0 con 0x7fc1380b17d0 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc13806e9f0 con 0x7fc1380b17d0 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200293236 0 0) 0x7fc124004eb0 con 0x7fc138142690 2024-03-29T10:56:35.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc13806ecd0 con 0x7fc138142690 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc12c001510 con 0x7fc138142690 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): dump: 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:35.913 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4010834175 0 0) 0x7fc13806ecd0 con 0x7fc138142690 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 >> v1:172.21.15.155:6790/0 conn(0x7fc1380b17d0 legacy=0x7fc13806f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.927+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 >> v1:172.21.15.155:6789/0 conn(0x7fc1381422b0 legacy=0x7fc1381eb810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 1 monclient: found mon.b 2024-03-29T10:56:35.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13806efb0 con 0x7fc138142690 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 10 monclient: _finish_auth 0 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc11ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 5 monclient: authenticate success, global_id 4384 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 10 monclient: _renew_subs 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13806fae0 con 0x7fc138142690 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 10 monclient: _renew_subs 2024-03-29T10:56:35.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc138150110 con 0x7fc138142690 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.929+0000 7fc13d8e0640 1 librados: init done 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc12c001aa0 con 0x7fc138142690 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc11ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc12c003890 con 0x7fc138142690 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc11ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc11ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc11ffff640 10 monclient: dump: 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:35.916 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:35.917 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:35.917 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:35.917 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:35.917 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:35.917 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:35.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc134ff9640 4 set_mon_vals no callback set 2024-03-29T10:56:35.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.930+0000 7fc13d8e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:35.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.931+0000 7fc13d8e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.931+0000 7fc13d8e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.931+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc118005080 con 0x7fc138142690 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.935+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fc12c019860 con 0x7fc138142690 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.936+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fc12c0837c0 con 0x7fc138142690 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.936+0000 7fc11ffff640 10 client.4384.objecter ms_dispatch 0x7fc138000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.936+0000 7fc11ffff640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.936+0000 7fc11ffff640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.936+0000 7fc11ffff640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.936+0000 7fc136ffd640 10 monclient: get_auth_request con 0x7fc12406dd80 auth_method 0 2024-03-29T10:56:35.919 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.937+0000 7fc11ffff640 10 client.4384.objecter ms_handle_connect 0x7fc12406dd80 2024-03-29T10:56:35.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.939+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc12c04eed0 con 0x7fc138142690 2024-03-29T10:56:35.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.939+0000 7fc11ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:35.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:35.939+0000 7fc11ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:36.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.274+0000 7fc13d8e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:36.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.274+0000 7fc13d8e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:36.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.274+0000 7fc13d8e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:56:36.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.274+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc118005330 con 0x7fc138142690 2024-03-29T10:56:36.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.276+0000 7fc11ffff640 1 -- 172.21.15.155:0/3927588537 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fc12c050fd0 con 0x7fc138142690 2024-03-29T10:56:36.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.276+0000 7fc11ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:36.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.276+0000 7fc11ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:36.258 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:36.258 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:36.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 10 librados: watch_flush enter 2024-03-29T10:56:36.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 10 librados: watch_flush exit 2024-03-29T10:56:36.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T10:56:36.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 10 client.4384.objecter successfully canceled tick 2024-03-29T10:56:36.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 >> v1:172.21.15.183:6816/1412921548 conn(0x7fc12406dd80 legacy=0x7fc124070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 10 monclient: shutdown 2024-03-29T10:56:36.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:36.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 >> v1:172.21.15.183:6789/0 conn(0x7fc138142690 legacy=0x7fc1381eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:36.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 shutdown_connections 2024-03-29T10:56:36.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.281+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 >> 172.21.15.155:0/3927588537 conn(0x7fc138066a00 msgr2=0x7fc138131810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:36.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.282+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 shutdown_connections 2024-03-29T10:56:36.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.282+0000 7fc13d8e0640 1 -- 172.21.15.155:0/3927588537 wait complete. 2024-03-29T10:56:36.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:36.282+0000 7fc13d8e0640 1 librados: shutdown 2024-03-29T10:56:36.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:43.088 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.263+0000 7fe902233640 10 monclient: get_monmap_and_config 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.263+0000 7fe902233640 10 monclient: build_initial_monmap 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.263+0000 7fe902233640 10 monclient: monmap: 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:43.264059+0000 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:43.264059+0000 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:43.244 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:43.245 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:43.245 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:43.245 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:43.245 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:43.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.264+0000 7fe902233640 1 Processor -- start 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.264+0000 7fe902233640 1 -- start start 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.264+0000 7fe902233640 10 monclient: init 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 10 monclient(hunting): picked mon.noname-b con 0x7fe8fc0b0770 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 10 monclient(hunting): picked mon.noname-c con 0x7fe8fc14a700 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 10 monclient(hunting): picked mon.noname-a con 0x7fe8fc0ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc0aba70 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc0aecb0 con 0x7fe8fc14a700 2024-03-29T10:56:43.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc0af150 con 0x7fe8fc0b0770 2024-03-29T10:56:43.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:43.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.265+0000 7fe902233640 10 monclient(hunting): authenticate will time out at 1106.557401s 2024-03-29T10:56:43.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.267+0000 7fe8fb7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe8fc0b0770 0x7fe8fc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:46943/0 (socket says 172.21.15.155:46943) 2024-03-29T10:56:43.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.267+0000 7fe8fb7fe640 1 -- 172.21.15.155:0/3173526816 learned_addr learned my addr 172.21.15.155:0/3173526816 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:43.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135340112 0 0) 0x7fe8fc0aba70 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:43.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:43.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:43.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:56:43.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e4003520 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656750848 0 0) 0x7fe8fc0aecb0 con 0x7fe8fc14a700 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8fc0aba70 con 0x7fe8fc14a700 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945598977 0 0) 0x7fe8fc0af150 con 0x7fe8fc0b0770 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.273+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8fc0aecb0 con 0x7fe8fc0b0770 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160324051 0 0) 0x7fe8e4003520 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 >> v1:172.21.15.155:6790/0 conn(0x7fe8fc0b0770 legacy=0x7fe8fc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 >> v1:172.21.15.183:6789/0 conn(0x7fe8fc14a700 legacy=0x7fe8fc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 monclient: found mon.noname-a 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc0ac9d0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe8ec001690 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: dump: 2024-03-29T10:56:43.260 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe902233640 5 monclient: authenticate success, global_id 4386 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe902233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8ec0017f0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:43.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 1 -- 172.21.15.155:0/3173526816 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe8ec001c00 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe8fa7fc640 10 monclient: dump: 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:43.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe902233640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.274+0000 7fe902233640 4 set_mon_vals no callback set 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.275+0000 7fe902233640 10 monclient: shutdown 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.275+0000 7fe902233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.275+0000 7fe902233640 1 -- 172.21.15.155:0/3173526816 >> v1:172.21.15.155:6789/0 conn(0x7fe8fc0ac5a0 legacy=0x7fe8fc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.275+0000 7fe902233640 1 -- 172.21.15.155:0/3173526816 shutdown_connections 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.275+0000 7fe902233640 1 -- 172.21.15.155:0/3173526816 >> 172.21.15.155:0/3173526816 conn(0x7fe8fc06b660 msgr2=0x7fe8fc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.275+0000 7fe902233640 1 -- 172.21.15.155:0/3173526816 shutdown_connections 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.275+0000 7fe902233640 1 -- 172.21.15.155:0/3173526816 wait complete. 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.276+0000 7fe902233640 10 monclient: build_initial_monmap 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.276+0000 7fe902233640 10 monclient: monmap: 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:43.263 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.277+0000 7fe902233640 1 librados: starting msgr at 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.277+0000 7fe902233640 1 librados: starting objecter 2024-03-29T10:56:43.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.277+0000 7fe902233640 1 Processor -- start 2024-03-29T10:56:43.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 1 -- start start 2024-03-29T10:56:43.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 1 librados: setting wanted keys 2024-03-29T10:56:43.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 1 librados: calling monclient init 2024-03-29T10:56:43.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient: init 2024-03-29T10:56:43.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:43.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:43.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient(hunting): picked mon.noname-c con 0x7fe8fc0ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient(hunting): picked mon.noname-a con 0x7fe8fc0b0770 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient(hunting): picked mon.noname-b con 0x7fe8fc14a700 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc071930 con 0x7fe8fc0b0770 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc06eae0 con 0x7fe8fc14a700 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc06deb0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.278+0000 7fe902233640 10 monclient(hunting): authenticate will time out at 1106.570839s 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.285+0000 7fe8fb7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe8fc0ac5a0 0x7fe8fc06e3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:47013/0 (socket says 172.21.15.155:47013) 2024-03-29T10:56:43.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.285+0000 7fe8fb7fe640 1 -- 172.21.15.155:0/2739838326 learned_addr learned my addr 172.21.15.155:0/2739838326 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:43.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.286+0000 7fe8e3fff640 10 client.?.objecter ms_handle_connect 0x7fe8fc0ac5a0 2024-03-29T10:56:43.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.286+0000 7fe8e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:43.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244978819 0 0) 0x7fe8fc06deb0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e8004eb0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 10 client.?.objecter ms_handle_connect 0x7fe8fc14a700 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.287+0000 7fe8e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 10 client.?.objecter ms_handle_connect 0x7fe8fc0b0770 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484258384 0 0) 0x7fe8fc06eae0 con 0x7fe8fc14a700 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:43.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.288+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8fc06deb0 con 0x7fe8fc14a700 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.289+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2635670871 0 0) 0x7fe8e8004eb0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.289+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8fc06eae0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.289+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe8f4001c90 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient(hunting): dump: 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:43.275 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3207851635 0 0) 0x7fe8fc06eae0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 >> v1:172.21.15.183:6789/0 conn(0x7fe8fc14a700 legacy=0x7fe8fc1e7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 >> v1:172.21.15.155:6789/0 conn(0x7fe8fc0b0770 legacy=0x7fe8fc143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 1 monclient: found mon.c 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc06d9d0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:56:43.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe8e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 10 monclient: _renew_subs 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc06dbe0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 10 monclient: _renew_subs 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc143b00 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.290+0000 7fe902233640 1 librados: init done 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.291+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8f4001df0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.291+0000 7fe8e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:43.277 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.291+0000 7fe8f97fa640 4 set_mon_vals no callback set 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe902233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe8f4004e10 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe902233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe902233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc005080 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe8e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe8e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.292+0000 7fe8e3fff640 10 monclient: dump: 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:43.278 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:43.279 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:43.279 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:43.279 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:43.279 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:43.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.296+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fe8f4004f70 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.296+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fe8f4084e10 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.297+0000 7fe8e3fff640 10 client.4424.objecter ms_dispatch 0x7fe8fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:43.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.297+0000 7fe8e3fff640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:43.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.297+0000 7fe8e3fff640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:43.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.297+0000 7fe8e3fff640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T10:56:43.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.300+0000 7fe8faffd640 10 monclient: get_auth_request con 0x7fe8e806dbb0 auth_method 0 2024-03-29T10:56:43.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.301+0000 7fe8e3fff640 10 client.4424.objecter ms_handle_connect 0x7fe8e806dbb0 2024-03-29T10:56:43.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.301+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8f40506f0 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.301+0000 7fe8e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:43.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.301+0000 7fe8e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:43.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.624+0000 7fe902233640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:43.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.624+0000 7fe902233640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:43.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.624+0000 7fe902233640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:43.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.624+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8dc005330 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.626+0000 7fe8e3fff640 1 -- 172.21.15.155:0/2739838326 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fe8f4052620 con 0x7fe8fc0ac5a0 2024-03-29T10:56:43.606 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.626+0000 7fe8e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:43.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.626+0000 7fe8e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:43.607 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:43.608 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:43.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.629+0000 7fe902233640 10 librados: watch_flush enter 2024-03-29T10:56:43.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.629+0000 7fe902233640 10 librados: watch_flush exit 2024-03-29T10:56:43.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.629+0000 7fe902233640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T10:56:43.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.629+0000 7fe902233640 10 client.4424.objecter successfully canceled tick 2024-03-29T10:56:43.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.632+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 >> v1:172.21.15.183:6816/1412921548 conn(0x7fe8e806dbb0 legacy=0x7fe8e8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 10 monclient: shutdown 2024-03-29T10:56:43.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:43.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 >> v1:172.21.15.155:6790/0 conn(0x7fe8fc0ac5a0 legacy=0x7fe8fc06e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:43.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 shutdown_connections 2024-03-29T10:56:43.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 >> 172.21.15.155:0/2739838326 conn(0x7fe8fc06b660 msgr2=0x7fe8fc0b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:43.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 shutdown_connections 2024-03-29T10:56:43.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 1 -- 172.21.15.155:0/2739838326 wait complete. 2024-03-29T10:56:43.614 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:43.633+0000 7fe902233640 1 librados: shutdown 2024-03-29T10:56:43.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:50.440 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:50.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.614+0000 7f379cea8640 10 monclient: get_monmap_and_config 2024-03-29T10:56:50.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.614+0000 7f379cea8640 10 monclient: build_initial_monmap 2024-03-29T10:56:50.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.614+0000 7f379cea8640 10 monclient: monmap: 2024-03-29T10:56:50.595 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:50.595 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:50.595 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:50.615117+0000 2024-03-29T10:56:50.595 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:50.615117+0000 2024-03-29T10:56:50.596 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:50.596 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:50.596 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:50.596 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:50.596 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:50.596 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:50.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.615+0000 7f379cea8640 1 Processor -- start 2024-03-29T10:56:50.597 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.615+0000 7f379cea8640 1 -- start start 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.615+0000 7f379cea8640 10 monclient: init 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 10 monclient(hunting): picked mon.noname-a con 0x7f37980a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 10 monclient(hunting): picked mon.noname-c con 0x7f379813ee90 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 10 monclient(hunting): picked mon.noname-b con 0x7f37980af900 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37980afd10 con 0x7f37980a9ac0 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37980af6f0 con 0x7f379813ee90 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37980aac80 con 0x7f37980af900 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:50.598 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.616+0000 7f379cea8640 10 monclient(hunting): authenticate will time out at 1113.908316s 2024-03-29T10:56:50.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.619+0000 7f3796d9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f37980af900 0x7f3798149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34133/0 (socket says 172.21.15.155:34133) 2024-03-29T10:56:50.600 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.619+0000 7f3796d9d640 1 -- 172.21.15.155:0/1286237989 learned_addr learned my addr 172.21.15.155:0/1286237989 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:50.601 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087223208 0 0) 0x7f37980aac80 con 0x7f37980af900 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780003520 con 0x7f37980af900 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3565387228 0 0) 0x7f3780003520 con 0x7f37980af900 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.620+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 >> v1:172.21.15.183:6789/0 conn(0x7f379813ee90 legacy=0x7f379813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 >> v1:172.21.15.155:6789/0 conn(0x7f37980a9ac0 legacy=0x7f37980adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 1 monclient: found mon.noname-b 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:56:50.602 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37980aaf60 con 0x7f37980af900 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 10 monclient: _finish_auth 0 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3790001a10 con 0x7f37980af900 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379559a640 10 monclient: dump: 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:50.603 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:50.604 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:50.604 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:50.604 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:50.604 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:50.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379cea8640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:56:50.604 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.621+0000 7f379cea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:50.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.624+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3790001be0 con 0x7f37980af900 2024-03-29T10:56:50.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.624+0000 7f379559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:50.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.624+0000 7f379559a640 1 -- 172.21.15.155:0/1286237989 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3790001ff0 con 0x7f37980af900 2024-03-29T10:56:50.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.624+0000 7f379559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:50.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.624+0000 7f379559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:50.605 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.624+0000 7f379559a640 10 monclient: dump: 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:50.606 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:50.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.625+0000 7f379cea8640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:50.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.625+0000 7f379cea8640 4 set_mon_vals no callback set 2024-03-29T10:56:50.607 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.626+0000 7f379cea8640 10 monclient: shutdown 2024-03-29T10:56:50.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.626+0000 7f379cea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:50.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.626+0000 7f379cea8640 1 -- 172.21.15.155:0/1286237989 >> v1:172.21.15.155:6790/0 conn(0x7f37980af900 legacy=0x7f3798149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:50.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.626+0000 7f379cea8640 1 -- 172.21.15.155:0/1286237989 shutdown_connections 2024-03-29T10:56:50.608 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.626+0000 7f379cea8640 1 -- 172.21.15.155:0/1286237989 >> 172.21.15.155:0/1286237989 conn(0x7f37980b35e0 msgr2=0x7f37980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:50.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.629+0000 7f379cea8640 1 -- 172.21.15.155:0/1286237989 shutdown_connections 2024-03-29T10:56:50.610 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.629+0000 7f379cea8640 1 -- 172.21.15.155:0/1286237989 wait complete. 2024-03-29T10:56:50.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.630+0000 7f379cea8640 10 monclient: build_initial_monmap 2024-03-29T10:56:50.611 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.630+0000 7f379cea8640 10 monclient: monmap: 2024-03-29T10:56:50.611 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:50.611 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.631+0000 7f379cea8640 1 librados: starting msgr at 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.631+0000 7f379cea8640 1 librados: starting objecter 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.631+0000 7f379cea8640 1 Processor -- start 2024-03-29T10:56:50.612 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.631+0000 7f379cea8640 1 -- start start 2024-03-29T10:56:50.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.631+0000 7f379cea8640 1 librados: setting wanted keys 2024-03-29T10:56:50.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.631+0000 7f379cea8640 1 librados: calling monclient init 2024-03-29T10:56:50.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.631+0000 7f379cea8640 10 monclient: init 2024-03-29T10:56:50.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.632+0000 7f379cea8640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:50.613 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.632+0000 7f379cea8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 10 monclient(hunting): picked mon.noname-c con 0x7f37980a9ac0 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 10 monclient(hunting): picked mon.noname-b con 0x7f37980af900 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 10 monclient(hunting): picked mon.noname-a con 0x7f379813ee90 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379813f5c0 con 0x7f379813ee90 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3798141570 con 0x7f37980af900 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3798142680 con 0x7f37980a9ac0 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:50.617 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.634+0000 7f379cea8640 10 monclient(hunting): authenticate will time out at 1113.926701s 2024-03-29T10:56:50.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.637+0000 7f3796d9d640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f379813ee90 0x7f37981e5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49665/0 (socket says 172.21.15.155:49665) 2024-03-29T10:56:50.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.637+0000 7f3796d9d640 1 -- 172.21.15.155:0/697381230 learned_addr learned my addr 172.21.15.155:0/697381230 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:50.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.638+0000 7f3786ffd640 10 client.?.objecter ms_handle_connect 0x7f379813ee90 2024-03-29T10:56:50.618 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.638+0000 7f3786ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:50.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 client.?.objecter ms_handle_connect 0x7f37980af900 2024-03-29T10:56:50.620 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:50.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 client.?.objecter ms_handle_connect 0x7f37980a9ac0 2024-03-29T10:56:50.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:50.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088374383 0 0) 0x7f3798142680 con 0x7f37980a9ac0 2024-03-29T10:56:50.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:50.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:50.621 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:50.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:56:50.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3778004eb0 con 0x7f37980a9ac0 2024-03-29T10:56:50.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772182696 0 0) 0x7f3798141570 con 0x7f37980af900 2024-03-29T10:56:50.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:50.622 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798142680 con 0x7f37980af900 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3377373849 0 0) 0x7f3778004eb0 con 0x7f37980a9ac0 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.639+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3798141570 con 0x7f37980a9ac0 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f37880018c0 con 0x7f37980a9ac0 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:50.623 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 10 monclient(hunting): dump: 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:50.624 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4161064281 0 0) 0x7f3798141570 con 0x7f37980a9ac0 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 >> v1:172.21.15.183:6789/0 conn(0x7f37980af900 legacy=0x7f37981db860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 >> v1:172.21.15.155:6789/0 conn(0x7f379813ee90 legacy=0x7f37981e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.640+0000 7f3786ffd640 1 monclient: found mon.c 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f3786ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f3786ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37980722e0 con 0x7f37980a9ac0 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f3786ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f3786ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f3786ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:50.625 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f379cea8640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:56:50.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.642+0000 7f379cea8640 10 monclient: _renew_subs 2024-03-29T10:56:50.626 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f379cea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:50.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379813a0e0 con 0x7f37980a9ac0 2024-03-29T10:56:50.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f379cea8640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:50.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f379cea8640 10 monclient: _renew_subs 2024-03-29T10:56:50.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f379cea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:50.630 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37981dc150 con 0x7f37980a9ac0 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f379cea8640 1 librados: init done 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37880035f0 con 0x7f37980a9ac0 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f3786ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3788003900 con 0x7f37980a9ac0 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f3786ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f3786ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.643+0000 7f3786ffd640 10 monclient: dump: 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:50.631 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.645+0000 7f379cea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.645+0000 7f379cea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.645+0000 7f379cea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.645+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3798075f10 con 0x7f37980a9ac0 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.645+0000 7f3787fff640 4 set_mon_vals no callback set 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.648+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f37880198d0 con 0x7f37980a9ac0 2024-03-29T10:56:50.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.649+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f3788083880 con 0x7f37980a9ac0 2024-03-29T10:56:50.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.649+0000 7f3786ffd640 10 client.4430.objecter ms_dispatch 0x7f3798000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:50.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.649+0000 7f3786ffd640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:50.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.649+0000 7f3786ffd640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:50.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.649+0000 7f3786ffd640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T10:56:50.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.650+0000 7f3795d9b640 10 monclient: get_auth_request con 0x7f377806dd70 auth_method 0 2024-03-29T10:56:50.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.653+0000 7f3786ffd640 10 client.4430.objecter ms_handle_connect 0x7f377806dd70 2024-03-29T10:56:50.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.659+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f378804f160 con 0x7f37980a9ac0 2024-03-29T10:56:50.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.660+0000 7f3786ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:50.640 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.660+0000 7f3786ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:50.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.989+0000 7f379cea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:50.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.989+0000 7f379cea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:50.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.989+0000 7f379cea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:50.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.989+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f379806ab70 con 0x7f37980a9ac0 2024-03-29T10:56:50.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.992+0000 7f3786ffd640 1 -- 172.21.15.155:0/697381230 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f3788051090 con 0x7f37980a9ac0 2024-03-29T10:56:50.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.993+0000 7f3786ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:50.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.993+0000 7f3786ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:50.973 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:50.973 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 10 librados: watch_flush enter 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 10 librados: watch_flush exit 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 10 client.4430.objecter successfully canceled tick 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 >> v1:172.21.15.183:6816/1412921548 conn(0x7f377806dd70 legacy=0x7f3778070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 10 monclient: shutdown 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:50.978 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 >> v1:172.21.15.155:6790/0 conn(0x7f37980a9ac0 legacy=0x7f3798141f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:50.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 shutdown_connections 2024-03-29T10:56:50.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.997+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 >> 172.21.15.155:0/697381230 conn(0x7f37980b35e0 msgr2=0x7f37980ad310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:50.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.998+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 shutdown_connections 2024-03-29T10:56:50.979 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:50.998+0000 7f379cea8640 1 -- 172.21.15.155:0/697381230 wait complete. 2024-03-29T10:56:50.980 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:51.000+0000 7f379cea8640 1 librados: shutdown 2024-03-29T10:56:50.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:57.805 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:57.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.976+0000 7ff4c90db640 10 monclient: get_monmap_and_config 2024-03-29T10:56:57.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.976+0000 7ff4c90db640 10 monclient: build_initial_monmap 2024-03-29T10:56:57.957 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.977+0000 7ff4c90db640 10 monclient: monmap: 2024-03-29T10:56:57.957 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:57.957 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:56:57.977674+0000 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:56:57.977674+0000 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:56:57.958 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:57.959 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.979+0000 7ff4c90db640 1 Processor -- start 2024-03-29T10:56:57.964 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.979+0000 7ff4c90db640 1 -- start start 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.979+0000 7ff4c90db640 10 monclient: init 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 10 monclient(hunting): picked mon.noname-a con 0x7ff4bc0389e0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 10 monclient(hunting): picked mon.noname-b con 0x7ff4bc038dc0 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 10 monclient(hunting): picked mon.noname-c con 0x7ff4bc030660 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc0c54c0 con 0x7ff4bc0389e0 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc036430 con 0x7ff4bc030660 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc039840 con 0x7ff4bc038dc0 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.980+0000 7ff4c90db640 10 monclient(hunting): authenticate will time out at 1121.272804s 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.984+0000 7ff4c3fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff4bc0389e0 0x7ff4bc033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:49869/0 (socket says 172.21.15.155:49869) 2024-03-29T10:56:57.965 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.984+0000 7ff4c3fff640 1 -- 172.21.15.155:0/899414378 learned_addr learned my addr 172.21.15.155:0/899414378 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:57.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.987+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785588040 0 0) 0x7ff4bc036430 con 0x7ff4bc030660 2024-03-29T10:56:57.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.987+0000 7ff4c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:57.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.987+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:57.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.987+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:57.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.987+0000 7ff4c2ffd640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:56:57.968 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.987+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b4003520 con 0x7ff4bc030660 2024-03-29T10:56:57.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912054810 0 0) 0x7ff4bc0c54c0 con 0x7ff4bc0389e0 2024-03-29T10:56:57.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:57.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:57.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:57.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:56:57.969 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc036430 con 0x7ff4bc0389e0 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790138137 0 0) 0x7ff4b4003520 con 0x7ff4bc030660 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 >> v1:172.21.15.155:6790/0 conn(0x7ff4bc038dc0 legacy=0x7ff4bc0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 >> v1:172.21.15.155:6789/0 conn(0x7ff4bc0389e0 legacy=0x7ff4bc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc035690 con 0x7ff4bc030660 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:57.970 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff4b8001610 con 0x7ff4bc030660 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.988+0000 7ff4c2ffd640 10 monclient: dump: 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:57.971 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c90db640 5 monclient: authenticate success, global_id 4396 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c90db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b80017e0 con 0x7ff4bc030660 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c2ffd640 1 -- 172.21.15.155:0/899414378 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff4b8001bf0 con 0x7ff4bc030660 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c2ffd640 10 monclient: dump: 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:57.972 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:57.973 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:57.973 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:57.973 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:57.973 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:57.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c90db640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:57.973 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.989+0000 7ff4c90db640 4 set_mon_vals no callback set 2024-03-29T10:56:57.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.990+0000 7ff4c90db640 10 monclient: shutdown 2024-03-29T10:56:57.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.990+0000 7ff4c90db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:57.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.990+0000 7ff4c90db640 1 -- 172.21.15.155:0/899414378 >> v1:172.21.15.183:6789/0 conn(0x7ff4bc030660 legacy=0x7ff4bc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:57.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.990+0000 7ff4c90db640 1 -- 172.21.15.155:0/899414378 shutdown_connections 2024-03-29T10:56:57.974 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.990+0000 7ff4c90db640 1 -- 172.21.15.155:0/899414378 >> 172.21.15.155:0/899414378 conn(0x7ff4bc005c00 msgr2=0x7ff4bc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.991+0000 7ff4c90db640 1 -- 172.21.15.155:0/899414378 shutdown_connections 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.991+0000 7ff4c90db640 1 -- 172.21.15.155:0/899414378 wait complete. 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.992+0000 7ff4c90db640 10 monclient: build_initial_monmap 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.992+0000 7ff4c90db640 10 monclient: monmap: 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:56:57.975 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:57.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.993+0000 7ff4c90db640 1 librados: starting msgr at 2024-03-29T10:56:57.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.993+0000 7ff4c90db640 1 librados: starting objecter 2024-03-29T10:56:57.976 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.993+0000 7ff4c90db640 1 Processor -- start 2024-03-29T10:56:57.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.995+0000 7ff4c90db640 1 -- start start 2024-03-29T10:56:57.982 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.995+0000 7ff4c90db640 1 librados: setting wanted keys 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.995+0000 7ff4c90db640 1 librados: calling monclient init 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.995+0000 7ff4c90db640 10 monclient: init 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 10 monclient(hunting): picked mon.noname-c con 0x7ff4bc030660 addr v1:172.21.15.155:6790/0 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 10 monclient(hunting): picked mon.noname-b con 0x7ff4bc0389e0 addr v1:172.21.15.183:6789/0 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 10 monclient(hunting): picked mon.noname-a con 0x7ff4bc0c73d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc02da90 con 0x7ff4bc0c73d0 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc0c83b0 con 0x7ff4bc0389e0 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc1605c0 con 0x7ff4bc030660 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:57.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:57.996+0000 7ff4c90db640 10 monclient(hunting): authenticate will time out at 1121.288563s 2024-03-29T10:56:57.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.001+0000 7ff4c37fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff4bc0389e0 0x7ff4bc0bf210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:47923/0 (socket says 172.21.15.155:47923) 2024-03-29T10:56:57.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.001+0000 7ff4c37fe640 1 -- 172.21.15.155:0/1632849060 learned_addr learned my addr 172.21.15.155:0/1632849060 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:56:57.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.002+0000 7ff4c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff4bc0389e0 2024-03-29T10:56:57.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.002+0000 7ff4c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:57.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.003+0000 7ff4c0ff9640 10 client.?.objecter ms_handle_connect 0x7ff4bc030660 2024-03-29T10:56:57.984 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.003+0000 7ff4c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898584210 0 0) 0x7ff4bc1605c0 con 0x7ff4bc030660 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a8004eb0 con 0x7ff4bc030660 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511311414 0 0) 0x7ff4bc0c83b0 con 0x7ff4bc0389e0 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:56:57.987 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.006+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc1605c0 con 0x7ff4bc0389e0 2024-03-29T10:56:57.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3328373657 0 0) 0x7ff4a8004eb0 con 0x7ff4bc030660 2024-03-29T10:56:57.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4bc0c83b0 con 0x7ff4bc030660 2024-03-29T10:56:57.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff4c404fdd0 con 0x7ff4bc030660 2024-03-29T10:56:57.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:57.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:57.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1819357946 0 0) 0x7ff4bc1605c0 con 0x7ff4bc0389e0 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4a8004eb0 con 0x7ff4bc0389e0 2024-03-29T10:56:57.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff4b0001910 con 0x7ff4bc0389e0 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:57.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 459345085 0 0) 0x7ff4bc0c83b0 con 0x7ff4bc030660 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 >> v1:172.21.15.183:6789/0 conn(0x7ff4bc0389e0 legacy=0x7ff4bc0bf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 >> v1:172.21.15.155:6789/0 conn(0x7ff4bc0c73d0 legacy=0x7ff4bc168230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 monclient: found mon.c 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc160750 con 0x7ff4bc030660 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 10 monclient: _renew_subs 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:57.991 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc0ca0f0 con 0x7ff4bc030660 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 10 monclient: _renew_subs 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc0c7870 con 0x7ff4bc030660 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.007+0000 7ff4c90db640 1 librados: init done 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.008+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4c404f400 con 0x7ff4bc030660 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.008+0000 7ff4c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.008+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff4c4075710 con 0x7ff4bc030660 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.008+0000 7ff4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.008+0000 7ff4c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.008+0000 7ff4c0ff9640 10 monclient: dump: 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:56:57.992 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:56:57.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.008+0000 7ff4c27fc640 4 set_mon_vals no callback set 2024-03-29T10:56:57.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.009+0000 7ff4c90db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:57.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.009+0000 7ff4c90db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:57.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.009+0000 7ff4c90db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:57.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.009+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a4005080 con 0x7ff4bc030660 2024-03-29T10:56:57.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.010+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7ff4c4067bf0 con 0x7ff4bc030660 2024-03-29T10:56:57.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.010+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7ff4c40ff500 con 0x7ff4bc030660 2024-03-29T10:56:57.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.010+0000 7ff4c0ff9640 10 client.4436.objecter ms_dispatch 0x7ff4c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:57.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.010+0000 7ff4c0ff9640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:57.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.010+0000 7ff4c0ff9640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:57.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.010+0000 7ff4c0ff9640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T10:56:57.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.011+0000 7ff4c37fe640 10 monclient: get_auth_request con 0x7ff4a806dbe0 auth_method 0 2024-03-29T10:56:57.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.014+0000 7ff4c0ff9640 10 client.4436.objecter ms_handle_connect 0x7ff4a806dbe0 2024-03-29T10:56:58.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.021+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4c40cade0 con 0x7ff4bc030660 2024-03-29T10:56:58.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.021+0000 7ff4c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:58.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.021+0000 7ff4c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:58.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.347+0000 7ff4c90db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:58.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.347+0000 7ff4c90db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:58.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.347+0000 7ff4c90db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:56:58.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.347+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4a4005330 con 0x7ff4bc030660 2024-03-29T10:56:58.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.348+0000 7ff4c0ff9640 1 -- 172.21.15.155:0/1632849060 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7ff4c4067ea0 con 0x7ff4bc030660 2024-03-29T10:56:58.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.348+0000 7ff4c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:58.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.348+0000 7ff4c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:58.330 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:56:58.330 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:58.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.353+0000 7ff4c90db640 10 librados: watch_flush enter 2024-03-29T10:56:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.353+0000 7ff4c90db640 10 librados: watch_flush exit 2024-03-29T10:56:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.353+0000 7ff4c90db640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T10:56:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.353+0000 7ff4c90db640 10 client.4436.objecter successfully canceled tick 2024-03-29T10:56:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.354+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 >> v1:172.21.15.183:6816/1412921548 conn(0x7ff4a806dbe0 legacy=0x7ff4a80700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.354+0000 7ff4c90db640 10 monclient: shutdown 2024-03-29T10:56:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.354+0000 7ff4c90db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:58.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.354+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 >> v1:172.21.15.155:6790/0 conn(0x7ff4bc030660 legacy=0x7ff4bc15feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:58.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.358+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 shutdown_connections 2024-03-29T10:56:58.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.359+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 >> 172.21.15.155:0/1632849060 conn(0x7ff4bc005c00 msgr2=0x7ff4bc0c4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:58.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.361+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 shutdown_connections 2024-03-29T10:56:58.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.361+0000 7ff4c90db640 1 -- 172.21.15.155:0/1632849060 wait complete. 2024-03-29T10:56:58.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:56:58.361+0000 7ff4c90db640 1 librados: shutdown 2024-03-29T10:56:58.363 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:05.170 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.344+0000 7fabf04fb640 10 monclient: get_monmap_and_config 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.344+0000 7fabf04fb640 10 monclient: build_initial_monmap 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.344+0000 7fabf04fb640 10 monclient: monmap: 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:05.345609+0000 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:05.345609+0000 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:05.325 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:05.326 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:05.326 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:05.326 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:05.326 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:05.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.346+0000 7fabf04fb640 1 Processor -- start 2024-03-29T10:57:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.346+0000 7fabf04fb640 1 -- start start 2024-03-29T10:57:05.327 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.346+0000 7fabf04fb640 10 monclient: init 2024-03-29T10:57:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 10 monclient(hunting): picked mon.noname-b con 0x7fabe81457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 10 monclient(hunting): picked mon.noname-c con 0x7fabe813f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 10 monclient(hunting): picked mon.noname-a con 0x7fabe80a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:05.328 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe80aaef0 con 0x7fabe80a9d70 2024-03-29T10:57:05.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe8145bd0 con 0x7fabe813f650 2024-03-29T10:57:05.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe8145eb0 con 0x7fabe81457f0 2024-03-29T10:57:05.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:05.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.347+0000 7fabf04fb640 10 monclient(hunting): authenticate will time out at 1128.639683s 2024-03-29T10:57:05.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.351+0000 7fabee297640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fabe81457f0 0x7fabe813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:38237/0 (socket says 172.21.15.155:38237) 2024-03-29T10:57:05.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.351+0000 7fabee297640 1 -- 172.21.15.155:0/3931201336 learned_addr learned my addr 172.21.15.155:0/3931201336 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053086709 0 0) 0x7fabe8145eb0 con 0x7fabe81457f0 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd4003520 con 0x7fabe81457f0 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329757297 0 0) 0x7fabe80aaef0 con 0x7fabe80a9d70 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:57:05.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8145eb0 con 0x7fabe80a9d70 2024-03-29T10:57:05.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1485605315 0 0) 0x7fabd4003520 con 0x7fabe81457f0 2024-03-29T10:57:05.335 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.352+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 >> v1:172.21.15.183:6789/0 conn(0x7fabe813f650 legacy=0x7fabe80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:05.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.354+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 >> v1:172.21.15.155:6789/0 conn(0x7fabe80a9d70 legacy=0x7fabe814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: _finish_hunting 0 2024-03-29T10:57:05.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 1 monclient: found mon.noname-b 2024-03-29T10:57:05.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:05.336 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe8146340 con 0x7fabe81457f0 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: _finish_auth 0 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: _check_auth_tickets 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabf04fb640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabf04fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fabd8001690 con 0x7fabe81457f0 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: dump: 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.337 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabf04fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd80017f0 con 0x7fabe81457f0 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabf04fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabf04fb640 4 set_mon_vals no callback set 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 1 -- 172.21.15.155:0/3931201336 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fabd8001c40 con 0x7fabe81457f0 2024-03-29T10:57:05.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.355+0000 7fabed295640 10 monclient: dump: 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.356+0000 7fabf04fb640 10 monclient: shutdown 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.356+0000 7fabf04fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:05.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.356+0000 7fabf04fb640 1 -- 172.21.15.155:0/3931201336 >> v1:172.21.15.155:6790/0 conn(0x7fabe81457f0 legacy=0x7fabe813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.356+0000 7fabf04fb640 1 -- 172.21.15.155:0/3931201336 shutdown_connections 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.356+0000 7fabf04fb640 1 -- 172.21.15.155:0/3931201336 >> 172.21.15.155:0/3931201336 conn(0x7fabe8066a00 msgr2=0x7fabe8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.357+0000 7fabf04fb640 1 -- 172.21.15.155:0/3931201336 shutdown_connections 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.357+0000 7fabf04fb640 1 -- 172.21.15.155:0/3931201336 wait complete. 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.358+0000 7fabf04fb640 10 monclient: build_initial_monmap 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.358+0000 7fabf04fb640 10 monclient: monmap: 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:05.342 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:05.343 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:05.343 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:05.343 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:05.343 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:05.343 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:05.343 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.358+0000 7fabf04fb640 1 librados: starting msgr at 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.358+0000 7fabf04fb640 1 librados: starting objecter 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.359+0000 7fabf04fb640 1 Processor -- start 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.359+0000 7fabf04fb640 1 -- start start 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.359+0000 7fabf04fb640 1 librados: setting wanted keys 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.359+0000 7fabf04fb640 1 librados: calling monclient init 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.359+0000 7fabf04fb640 10 monclient: init 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 10 monclient(hunting): picked mon.noname-b con 0x7fabe80a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:05.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 10 monclient(hunting): picked mon.noname-c con 0x7fabe813f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:05.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 10 monclient(hunting): picked mon.noname-a con 0x7fabe81457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:05.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe8141810 con 0x7fabe81457f0 2024-03-29T10:57:05.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe8141a80 con 0x7fabe80a9d70 2024-03-29T10:57:05.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe8144e80 con 0x7fabe813f650 2024-03-29T10:57:05.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.360+0000 7fabf04fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:05.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.361+0000 7fabf04fb640 10 monclient(hunting): authenticate will time out at 1128.652894s 2024-03-29T10:57:05.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.366+0000 7fabeea98640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fabe81457f0 0x7fabe81425b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:51365/0 (socket says 172.21.15.155:51365) 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.366+0000 7fabeea98640 1 -- 172.21.15.155:0/3844353359 learned_addr learned my addr 172.21.15.155:0/3844353359 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 client.?.objecter ms_handle_connect 0x7fabe81457f0 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 client.?.objecter ms_handle_connect 0x7fabe80a9d70 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310604221 0 0) 0x7fabe8141810 con 0x7fabe81457f0 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc004eb0 con 0x7fabe81457f0 2024-03-29T10:57:05.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817411991 0 0) 0x7fabe8141a80 con 0x7fabe80a9d70 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe8141810 con 0x7fabe80a9d70 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.367+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47291355 0 0) 0x7fabe8141810 con 0x7fabe80a9d70 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.368+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe8141a80 con 0x7fabe80a9d70 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.368+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fabd8001b40 con 0x7fabe80a9d70 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.368+0000 7fabdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.368+0000 7fabdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.368+0000 7fabdeffd640 10 monclient(hunting): dump: 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:05.354 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.368+0000 7fabdeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.369+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2176296624 0 0) 0x7fabcc004eb0 con 0x7fabe81457f0 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.369+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe8141810 con 0x7fabe81457f0 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.369+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fabe0001dd0 con 0x7fabe81457f0 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.369+0000 7fabdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.355 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.369+0000 7fabdeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.369+0000 7fabdeffd640 10 monclient(hunting): dump: 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.369+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2143916936 0 0) 0x7fabe8141a80 con 0x7fabe80a9d70 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 >> v1:172.21.15.155:6790/0 conn(0x7fabe813f650 legacy=0x7fabe8141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 >> v1:172.21.15.155:6789/0 conn(0x7fabe81457f0 legacy=0x7fabe81425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 1 monclient: found mon.b 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe8144aa0 con 0x7fabe80a9d70 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabf04fb640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.370+0000 7fabf04fb640 10 monclient: _renew_subs 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.371+0000 7fabf04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.371+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe8135750 con 0x7fabe80a9d70 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.371+0000 7fabf04fb640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.371+0000 7fabf04fb640 10 monclient: _renew_subs 2024-03-29T10:57:05.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.371+0000 7fabf04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:05.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.371+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe8140440 con 0x7fabe80a9d70 2024-03-29T10:57:05.358 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.371+0000 7fabf04fb640 1 librados: init done 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.374+0000 7fabf04fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.374+0000 7fabf04fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.374+0000 7fabf04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.374+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe8145010 con 0x7fabe80a9d70 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.374+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd8004a90 con 0x7fabe80a9d70 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.374+0000 7fabdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.374+0000 7fabeca94640 4 set_mon_vals no callback set 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.376+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fabd8001d80 con 0x7fabe80a9d70 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.376+0000 7fabdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.376+0000 7fabdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.376+0000 7fabdeffd640 10 monclient: dump: 2024-03-29T10:57:05.360 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.376+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fabd801ac10 con 0x7fabe80a9d70 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.377+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fabd8084a60 con 0x7fabe80a9d70 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.377+0000 7fabdeffd640 10 client.4402.objecter ms_dispatch 0x7fabe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.377+0000 7fabdeffd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:05.361 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.377+0000 7fabdeffd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:05.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.377+0000 7fabdeffd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T10:57:05.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.378+0000 7fabeda96640 10 monclient: get_auth_request con 0x7fabcc06de40 auth_method 0 2024-03-29T10:57:05.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.379+0000 7fabdeffd640 10 client.4402.objecter ms_handle_connect 0x7fabcc06de40 2024-03-29T10:57:05.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.384+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabd8050170 con 0x7fabe80a9d70 2024-03-29T10:57:05.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.384+0000 7fabdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:05.365 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.384+0000 7fabdeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:05.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.711+0000 7fabf04fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:05.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.711+0000 7fabf04fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:05.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.711+0000 7fabf04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:05.692 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.711+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabe8075f10 con 0x7fabe80a9d70 2024-03-29T10:57:05.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.715+0000 7fabdeffd640 1 -- 172.21.15.155:0/3844353359 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fabd8052270 con 0x7fabe80a9d70 2024-03-29T10:57:05.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.715+0000 7fabdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:05.695 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.715+0000 7fabdeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:05.696 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:05.696 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:05.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 10 librados: watch_flush enter 2024-03-29T10:57:05.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 10 librados: watch_flush exit 2024-03-29T10:57:05.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T10:57:05.700 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 10 client.4402.objecter successfully canceled tick 2024-03-29T10:57:05.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 >> v1:172.21.15.183:6816/1412921548 conn(0x7fabcc06de40 legacy=0x7fabcc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 10 monclient: shutdown 2024-03-29T10:57:05.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:05.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 >> v1:172.21.15.183:6789/0 conn(0x7fabe80a9d70 legacy=0x7fabe8135040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:05.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 shutdown_connections 2024-03-29T10:57:05.701 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.720+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 >> 172.21.15.155:0/3844353359 conn(0x7fabe8066a00 msgr2=0x7fabe80b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:05.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.722+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 shutdown_connections 2024-03-29T10:57:05.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.722+0000 7fabf04fb640 1 -- 172.21.15.155:0/3844353359 wait complete. 2024-03-29T10:57:05.702 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:05.722+0000 7fabf04fb640 1 librados: shutdown 2024-03-29T10:57:05.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:12.527 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:12.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.720+0000 7f7ba84a8640 10 monclient: get_monmap_and_config 2024-03-29T10:57:12.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.720+0000 7f7ba84a8640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.706 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.720+0000 7f7ba84a8640 10 monclient: monmap: 2024-03-29T10:57:12.706 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:12.706 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:12.706 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:12.721873+0000 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:12.721873+0000 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.721+0000 7f7ba84a8640 1 Processor -- start 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.721+0000 7f7ba84a8640 1 -- start start 2024-03-29T10:57:12.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.721+0000 7f7ba84a8640 10 monclient: init 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba00ad6c0 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba00af6c0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba014fdc0 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba00a94e0 con 0x7f7ba00ad6c0 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba00af3a0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba00afb60 con 0x7f7ba00af6c0 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.722+0000 7f7ba84a8640 10 monclient(hunting): authenticate will time out at 1136.015633s 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.725+0000 7f7ba6a45640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7ba014fdc0 0x7f7ba015a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:58929/0 (socket says 172.21.15.155:58929) 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.725+0000 7f7ba6a45640 1 -- 172.21.15.155:0/1721734193 learned_addr learned my addr 172.21.15.155:0/1721734193 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.726+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248564623 0 0) 0x7f7ba00af3a0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.726+0000 7f7ba5242640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.726+0000 7f7ba5242640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.726+0000 7f7ba5242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.726+0000 7f7ba5242640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:57:12.710 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.726+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b8c003520 con 0x7f7ba014fdc0 2024-03-29T10:57:12.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.729+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 718427136 0 0) 0x7f7b8c003520 con 0x7f7ba014fdc0 2024-03-29T10:57:12.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 >> v1:172.21.15.155:6790/0 conn(0x7f7ba00af6c0 legacy=0x7f7ba0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:12.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 >> v1:172.21.15.155:6789/0 conn(0x7f7ba00ad6c0 legacy=0x7f7ba014f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:12.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 1 monclient: found mon.noname-c 2024-03-29T10:57:12.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba00aab80 con 0x7f7ba014fdc0 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.730+0000 7f7ba5242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.731+0000 7f7ba84a8640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.731+0000 7f7ba84a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.731+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7b900012f0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.731+0000 7f7ba5242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.731+0000 7f7ba5242640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:12.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.731+0000 7f7ba5242640 10 monclient: dump: 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:12.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.731+0000 7f7ba84a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:12.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b90001580 con 0x7f7ba014fdc0 2024-03-29T10:57:12.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba5242640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba5242640 1 -- 172.21.15.155:0/1721734193 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7b90001910 con 0x7f7ba014fdc0 2024-03-29T10:57:12.716 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba5242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba5242640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba5242640 10 monclient: dump: 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:12.717 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba84a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.733+0000 7f7ba84a8640 4 set_mon_vals no callback set 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.734+0000 7f7ba84a8640 10 monclient: shutdown 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.734+0000 7f7ba84a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.734+0000 7f7ba84a8640 1 -- 172.21.15.155:0/1721734193 >> v1:172.21.15.183:6789/0 conn(0x7f7ba014fdc0 legacy=0x7f7ba015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.734+0000 7f7ba84a8640 1 -- 172.21.15.155:0/1721734193 shutdown_connections 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.734+0000 7f7ba84a8640 1 -- 172.21.15.155:0/1721734193 >> 172.21.15.155:0/1721734193 conn(0x7f7ba00b35e0 msgr2=0x7f7ba00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.734+0000 7f7ba84a8640 1 -- 172.21.15.155:0/1721734193 shutdown_connections 2024-03-29T10:57:12.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.735+0000 7f7ba84a8640 1 -- 172.21.15.155:0/1721734193 wait complete. 2024-03-29T10:57:12.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.736+0000 7f7ba84a8640 10 monclient: build_initial_monmap 2024-03-29T10:57:12.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.736+0000 7f7ba84a8640 10 monclient: monmap: 2024-03-29T10:57:12.720 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:12.720 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:12.720 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:12.720 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.736+0000 7f7ba84a8640 1 librados: starting msgr at 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.737+0000 7f7ba84a8640 1 librados: starting objecter 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.737+0000 7f7ba84a8640 1 Processor -- start 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.737+0000 7f7ba84a8640 1 -- start start 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.737+0000 7f7ba84a8640 1 librados: setting wanted keys 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.737+0000 7f7ba84a8640 1 librados: calling monclient init 2024-03-29T10:57:12.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.737+0000 7f7ba84a8640 10 monclient: init 2024-03-29T10:57:12.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.738+0000 7f7ba84a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:12.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.738+0000 7f7ba84a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:12.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.738+0000 7f7ba84a8640 10 monclient(hunting): picked mon.noname-b con 0x7f7ba00ad6c0 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:12.722 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.738+0000 7f7ba84a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7ba00af6c0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba84a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7ba014fdc0 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba84a8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba0072260 con 0x7f7ba014fdc0 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba6a45640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7ba014fdc0 0x7f7ba014c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:60615/0 (socket says 172.21.15.155:60615) 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba6a45640 1 -- 172.21.15.155:0/2319104749 learned_addr learned my addr 172.21.15.155:0/2319104749 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba00724b0 con 0x7f7ba00ad6c0 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba00714e0 con 0x7f7ba00af6c0 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba84a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:12.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.742+0000 7f7ba84a8640 10 monclient(hunting): authenticate will time out at 1136.035830s 2024-03-29T10:57:12.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ba014fdc0 2024-03-29T10:57:12.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104586479 0 0) 0x7f7ba0072260 con 0x7f7ba014fdc0 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.744+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b84004eb0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.745+0000 7f7b96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ba00ad6c0 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.745+0000 7f7b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.745+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669067468 0 0) 0x7f7ba00724b0 con 0x7f7ba00ad6c0 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.746+0000 7f7b96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.746+0000 7f7b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:12.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.746+0000 7f7b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.746+0000 7f7b96ffd640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.746+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba0072260 con 0x7f7ba00ad6c0 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.746+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358196192 0 0) 0x7f7b84004eb0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.746+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ba00724b0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7b90001890 con 0x7f7ba014fdc0 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.731 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1521243542 0 0) 0x7f7ba0072260 con 0x7f7ba00ad6c0 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b84004eb0 con 0x7f7ba00ad6c0 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.747+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7b9c001830 con 0x7f7ba00ad6c0 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.748+0000 7f7b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.748+0000 7f7b96ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:12.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.748+0000 7f7b96ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.748+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 532257220 0 0) 0x7f7ba00724b0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.733 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.748+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 >> v1:172.21.15.155:6790/0 conn(0x7f7ba00af6c0 legacy=0x7f7ba014bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 >> v1:172.21.15.183:6789/0 conn(0x7f7ba00ad6c0 legacy=0x7f7ba0070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 1 monclient: found mon.a 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba0071670 con 0x7f7ba014fdc0 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:12.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7ba84a8640 5 monclient: authenticate success, global_id 4404 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7ba84a8640 10 monclient: _renew_subs 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7ba84a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.750+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba01e70a0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.751+0000 7f7ba84a8640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.751+0000 7f7ba84a8640 10 monclient: _renew_subs 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.751+0000 7f7ba84a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:12.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.751+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba0151df0 con 0x7f7ba014fdc0 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.751+0000 7f7ba84a8640 1 librados: init done 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.752+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b90001d60 con 0x7f7ba014fdc0 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.752+0000 7f7b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.752+0000 7f7ba4a41640 4 set_mon_vals no callback set 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.752+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7b90001370 con 0x7f7ba014fdc0 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.753+0000 7f7b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.753+0000 7f7b96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.753+0000 7f7b96ffd640 10 monclient: dump: 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.740 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.754+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7b9001ac40 con 0x7f7ba014fdc0 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.755+0000 7f7ba84a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.755+0000 7f7ba84a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.755+0000 7f7ba84a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.755+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b88005080 con 0x7f7ba014fdc0 2024-03-29T10:57:12.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.755+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f7b90084a50 con 0x7f7ba014fdc0 2024-03-29T10:57:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.755+0000 7f7b96ffd640 10 client.4404.objecter ms_dispatch 0x7f7ba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.756+0000 7f7b96ffd640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.756+0000 7f7ba6244640 10 monclient: get_auth_request con 0x7f7b8406dee0 auth_method 0 2024-03-29T10:57:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.756+0000 7f7b96ffd640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.756+0000 7f7b96ffd640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T10:57:12.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.756+0000 7f7b96ffd640 10 client.4404.objecter ms_handle_connect 0x7f7b8406dee0 2024-03-29T10:57:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.770+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b90050330 con 0x7f7ba014fdc0 2024-03-29T10:57:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.770+0000 7f7b96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:12.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:12.770+0000 7f7b96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:13.090 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.109+0000 7f7ba84a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.109+0000 7f7ba84a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.109+0000 7f7ba84a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:13.091 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.109+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b88005330 con 0x7f7ba014fdc0 2024-03-29T10:57:13.092 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:13.092 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:13.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.110+0000 7f7b96ffd640 1 -- 172.21.15.155:0/2319104749 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f7b90052260 con 0x7f7ba014fdc0 2024-03-29T10:57:13.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.110+0000 7f7b96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:13.093 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.110+0000 7f7b96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 10 librados: watch_flush enter 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 10 librados: watch_flush exit 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 10 client.4404.objecter successfully canceled tick 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 >> v1:172.21.15.183:6816/1412921548 conn(0x7f7b8406dee0 legacy=0x7f7b840703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 10 monclient: shutdown 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 >> v1:172.21.15.155:6789/0 conn(0x7f7ba014fdc0 legacy=0x7f7ba014c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:13.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 shutdown_connections 2024-03-29T10:57:13.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 >> 172.21.15.155:0/2319104749 conn(0x7f7ba00b35e0 msgr2=0x7f7ba00a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:13.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.114+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 shutdown_connections 2024-03-29T10:57:13.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.115+0000 7f7ba84a8640 1 -- 172.21.15.155:0/2319104749 wait complete. 2024-03-29T10:57:13.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:13.115+0000 7f7ba84a8640 1 librados: shutdown 2024-03-29T10:57:13.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:19.924 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:20.088 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.106+0000 7f4169397640 10 monclient: get_monmap_and_config 2024-03-29T10:57:20.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.106+0000 7f4169397640 10 monclient: build_initial_monmap 2024-03-29T10:57:20.089 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.106+0000 7f4169397640 10 monclient: monmap: 2024-03-29T10:57:20.089 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:20.089 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:20.089 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:20.108210+0000 2024-03-29T10:57:20.089 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:20.108210+0000 2024-03-29T10:57:20.089 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:20.090 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:20.090 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:20.090 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:20.090 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:20.090 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:20.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.111+0000 7f4169397640 1 Processor -- start 2024-03-29T10:57:20.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.111+0000 7f4169397640 1 -- start start 2024-03-29T10:57:20.092 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.111+0000 7f4169397640 10 monclient: init 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 10 monclient(hunting): picked mon.noname-a con 0x7f41640a9930 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 10 monclient(hunting): picked mon.noname-b con 0x7f4164149dc0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 10 monclient(hunting): picked mon.noname-c con 0x7f416414ff40 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41640aae60 con 0x7f41640a9930 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41640b4310 con 0x7f416414ff40 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41640afdd0 con 0x7f4164149dc0 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:20.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.112+0000 7f4169397640 10 monclient(hunting): authenticate will time out at 1143.405764s 2024-03-29T10:57:20.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.115+0000 7f41627fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f4164149dc0 0x7f416414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36382/0 (socket says 172.21.15.155:36382) 2024-03-29T10:57:20.097 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.115+0000 7f41627fc640 1 -- 172.21.15.155:0/2261964516 learned_addr learned my addr 172.21.15.155:0/2261964516 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:20.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.117+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991355861 0 0) 0x7f41640aae60 con 0x7f41640a9930 2024-03-29T10:57:20.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.117+0000 7f4161ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.117+0000 7f4161ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.117+0000 7f4161ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.117+0000 7f4161ffb640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:57:20.099 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.117+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f414c003520 con 0x7f41640a9930 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286634346 0 0) 0x7f41640afdd0 con 0x7f4164149dc0 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41640aae60 con 0x7f4164149dc0 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3103153686 0 0) 0x7f414c003520 con 0x7f41640a9930 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 >> v1:172.21.15.155:6790/0 conn(0x7f4164149dc0 legacy=0x7f416414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 >> v1:172.21.15.183:6789/0 conn(0x7f416414ff40 legacy=0x7f4164152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:20.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:20.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 monclient: found mon.noname-a 2024-03-29T10:57:20.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416414a950 con 0x7f41640a9930 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4154001690 con 0x7f41640a9930 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: dump: 2024-03-29T10:57:20.104 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4154001870 con 0x7f41640a9930 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 1 -- 172.21.15.155:0/2261964516 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4154001690 con 0x7f41640a9930 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.119+0000 7f4161ffb640 10 monclient: dump: 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.120+0000 7f4169397640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:57:20.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.120+0000 7f4169397640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.120+0000 7f4169397640 4 set_mon_vals no callback set 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.121+0000 7f4169397640 10 monclient: shutdown 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.121+0000 7f4169397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.121+0000 7f4169397640 1 -- 172.21.15.155:0/2261964516 >> v1:172.21.15.155:6789/0 conn(0x7f41640a9930 legacy=0x7f4164143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.121+0000 7f4169397640 1 -- 172.21.15.155:0/2261964516 shutdown_connections 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.121+0000 7f4169397640 1 -- 172.21.15.155:0/2261964516 >> 172.21.15.155:0/2261964516 conn(0x7f41640b35e0 msgr2=0x7f41640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.122+0000 7f4169397640 1 -- 172.21.15.155:0/2261964516 shutdown_connections 2024-03-29T10:57:20.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.122+0000 7f4169397640 1 -- 172.21.15.155:0/2261964516 wait complete. 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.123+0000 7f4169397640 10 monclient: build_initial_monmap 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.123+0000 7f4169397640 10 monclient: monmap: 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:20.108 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:20.109 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:20.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.124+0000 7f4169397640 1 librados: starting msgr at 2024-03-29T10:57:20.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.124+0000 7f4169397640 1 librados: starting objecter 2024-03-29T10:57:20.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.124+0000 7f4169397640 1 Processor -- start 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.127+0000 7f4169397640 1 -- start start 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.127+0000 7f4169397640 1 librados: setting wanted keys 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.127+0000 7f4169397640 1 librados: calling monclient init 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.127+0000 7f4169397640 10 monclient: init 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.128+0000 7f4169397640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.128+0000 7f4169397640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 10 monclient(hunting): picked mon.noname-a con 0x7f41640a9930 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 10 monclient(hunting): picked mon.noname-c con 0x7f4164149dc0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 10 monclient(hunting): picked mon.noname-b con 0x7f416414ff40 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:20.114 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416413da90 con 0x7f41640a9930 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416413dc20 con 0x7f416414ff40 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416413ddb0 con 0x7f4164149dc0 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f4169397640 10 monclient(hunting): authenticate will time out at 1143.424120s 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f41627fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f4164149dc0 0x7f41641db4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40466/0 (socket says 172.21.15.155:40466) 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f41627fc640 1 -- 172.21.15.155:0/162263230 learned_addr learned my addr 172.21.15.155:0/162263230 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f414b7fe640 10 client.?.objecter ms_handle_connect 0x7f4164149dc0 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.131+0000 7f414b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.132+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311513854 0 0) 0x7f416413ddb0 con 0x7f4164149dc0 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.132+0000 7f414b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.132+0000 7f414b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.132+0000 7f414b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.132+0000 7f414b7fe640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.132+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4150004eb0 con 0x7f4164149dc0 2024-03-29T10:57:20.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.133+0000 7f414b7fe640 10 client.?.objecter ms_handle_connect 0x7f41640a9930 2024-03-29T10:57:20.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.133+0000 7f414b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961981318 0 0) 0x7f416413da90 con 0x7f41640a9930 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f416413ddb0 con 0x7f41640a9930 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189411793 0 0) 0x7f4150004eb0 con 0x7f4164149dc0 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f416413da90 con 0x7f4164149dc0 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4158001910 con 0x7f4164149dc0 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:20.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:20.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.135+0000 7f414b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:20.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 10 client.?.objecter ms_handle_connect 0x7f416414ff40 2024-03-29T10:57:20.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:20.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212674649 0 0) 0x7f416413dc20 con 0x7f416414ff40 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4150004eb0 con 0x7f416414ff40 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357382990 0 0) 0x7f4150004eb0 con 0x7f416414ff40 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f416413dc20 con 0x7f416414ff40 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f415c0019d0 con 0x7f416414ff40 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.137+0000 7f414b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:20.120 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1106960634 0 0) 0x7f416413da90 con 0x7f4164149dc0 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 >> v1:172.21.15.183:6789/0 conn(0x7f416414ff40 legacy=0x7f4164145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 >> v1:172.21.15.155:6789/0 conn(0x7f41640a9930 legacy=0x7f416413d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 1 monclient: found mon.c 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:20.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41641dbc00 con 0x7f4164149dc0 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f414b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 10 monclient: _renew_subs 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41641dbe70 con 0x7f4164149dc0 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 10 monclient: _renew_subs 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f416413fad0 con 0x7f4164149dc0 2024-03-29T10:57:20.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.138+0000 7f4169397640 1 librados: init done 2024-03-29T10:57:20.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.139+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4158001a70 con 0x7f4164149dc0 2024-03-29T10:57:20.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.139+0000 7f414b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:20.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.139+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4158004a90 con 0x7f4164149dc0 2024-03-29T10:57:20.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.139+0000 7f414b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:20.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.139+0000 7f414b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:20.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.139+0000 7f414b7fe640 10 monclient: dump: 2024-03-29T10:57:20.123 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.140+0000 7f4169397640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.140+0000 7f4169397640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.140+0000 7f4169397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:20.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.140+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4164075f10 con 0x7f4164149dc0 2024-03-29T10:57:20.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.142+0000 7f41617fa640 4 set_mon_vals no callback set 2024-03-29T10:57:20.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.142+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f415801aa60 con 0x7f4164149dc0 2024-03-29T10:57:20.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.143+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f4158084a80 con 0x7f4164149dc0 2024-03-29T10:57:20.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.143+0000 7f414b7fe640 10 client.4445.objecter ms_dispatch 0x7f4164000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:20.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.143+0000 7f414b7fe640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:20.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.143+0000 7f414b7fe640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:20.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.143+0000 7f414b7fe640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T10:57:20.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.143+0000 7f4162ffd640 10 monclient: get_auth_request con 0x7f415006df00 auth_method 0 2024-03-29T10:57:20.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.149+0000 7f414b7fe640 10 client.4445.objecter ms_handle_connect 0x7f415006df00 2024-03-29T10:57:20.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.150+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4158050190 con 0x7f4164149dc0 2024-03-29T10:57:20.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.150+0000 7f414b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:20.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.150+0000 7f414b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:20.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.510+0000 7f4169397640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:20.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.510+0000 7f4169397640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:20.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.510+0000 7f4169397640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:20.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.510+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f416411f3b0 con 0x7f4164149dc0 2024-03-29T10:57:20.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.513+0000 7f414b7fe640 1 -- 172.21.15.155:0/162263230 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f4158052290 con 0x7f4164149dc0 2024-03-29T10:57:20.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.513+0000 7f414b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:20.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.513+0000 7f414b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:20.496 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:20.496 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:20.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.518+0000 7f4169397640 10 librados: watch_flush enter 2024-03-29T10:57:20.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.518+0000 7f4169397640 10 librados: watch_flush exit 2024-03-29T10:57:20.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.518+0000 7f4169397640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T10:57:20.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.518+0000 7f4169397640 10 client.4445.objecter successfully canceled tick 2024-03-29T10:57:20.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.519+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 >> v1:172.21.15.183:6816/1412921548 conn(0x7f415006df00 legacy=0x7f41500703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.519+0000 7f4169397640 10 monclient: shutdown 2024-03-29T10:57:20.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.519+0000 7f4169397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:20.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.519+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 >> v1:172.21.15.155:6790/0 conn(0x7f4164149dc0 legacy=0x7f41641db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:20.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.522+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 shutdown_connections 2024-03-29T10:57:20.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.522+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 >> 172.21.15.155:0/162263230 conn(0x7f41640b35e0 msgr2=0x7f41640ad530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:20.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.525+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 shutdown_connections 2024-03-29T10:57:20.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.525+0000 7f4169397640 1 -- 172.21.15.155:0/162263230 wait complete. 2024-03-29T10:57:20.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:20.525+0000 7f4169397640 1 librados: shutdown 2024-03-29T10:57:20.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:27.330 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:27.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.503+0000 7fa78159a640 10 monclient: get_monmap_and_config 2024-03-29T10:57:27.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.503+0000 7fa78159a640 10 monclient: build_initial_monmap 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.503+0000 7fa78159a640 10 monclient: monmap: 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:27.504189+0000 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:27.504189+0000 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:27.484 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:27.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.504+0000 7fa78159a640 1 Processor -- start 2024-03-29T10:57:27.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.504+0000 7fa78159a640 1 -- start start 2024-03-29T10:57:27.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.504+0000 7fa78159a640 10 monclient: init 2024-03-29T10:57:27.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 10 monclient(hunting): picked mon.noname-b con 0x7fa77c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 10 monclient(hunting): picked mon.noname-c con 0x7fa77c13f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 10 monclient(hunting): picked mon.noname-a con 0x7fa77c0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0aaef0 con 0x7fa77c0a9d70 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c145bd0 con 0x7fa77c13f650 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c145eb0 con 0x7fa77c1457f0 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa78159a640 10 monclient(hunting): authenticate will time out at 1150.797597s 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa77affd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa77c1457f0 0x7fa77c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36266/0 (socket says 172.21.15.155:36266) 2024-03-29T10:57:27.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.505+0000 7fa77affd640 1 -- 172.21.15.155:0/3337389678 learned_addr learned my addr 172.21.15.155:0/3337389678 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:27.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.506+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869554974 0 0) 0x7fa77c0aaef0 con 0x7fa77c0a9d70 2024-03-29T10:57:27.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.506+0000 7fa779ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:27.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.506+0000 7fa779ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:27.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.506+0000 7fa779ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.506+0000 7fa779ffb640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.506+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa768003520 con 0x7fa77c0a9d70 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4018224043 0 0) 0x7fa768003520 con 0x7fa77c0a9d70 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 >> v1:172.21.15.155:6790/0 conn(0x7fa77c1457f0 legacy=0x7fa77c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 >> v1:172.21.15.183:6789/0 conn(0x7fa77c13f650 legacy=0x7fa77c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 1 monclient: found mon.noname-a 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c146340 con 0x7fa77c0a9d70 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:27.488 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa78159a640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa78159a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa774001690 con 0x7fa77c0a9d70 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa779ffb640 10 monclient: dump: 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:27.489 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:27.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.507+0000 7fa78159a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7740018b0 con 0x7fa77c0a9d70 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa779ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa78159a640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa78159a640 4 set_mon_vals no callback set 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa779ffb640 1 -- 172.21.15.155:0/3337389678 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa774001c40 con 0x7fa77c0a9d70 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa779ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.508+0000 7fa779ffb640 10 monclient: dump: 2024-03-29T10:57:27.491 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.509+0000 7fa78159a640 10 monclient: shutdown 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.509+0000 7fa78159a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.509+0000 7fa78159a640 1 -- 172.21.15.155:0/3337389678 >> v1:172.21.15.155:6789/0 conn(0x7fa77c0a9d70 legacy=0x7fa77c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.509+0000 7fa78159a640 1 -- 172.21.15.155:0/3337389678 shutdown_connections 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.509+0000 7fa78159a640 1 -- 172.21.15.155:0/3337389678 >> 172.21.15.155:0/3337389678 conn(0x7fa77c066a00 msgr2=0x7fa77c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:27.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.509+0000 7fa78159a640 1 -- 172.21.15.155:0/3337389678 shutdown_connections 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.509+0000 7fa78159a640 1 -- 172.21.15.155:0/3337389678 wait complete. 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.510+0000 7fa78159a640 10 monclient: build_initial_monmap 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.510+0000 7fa78159a640 10 monclient: monmap: 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.510+0000 7fa78159a640 1 librados: starting msgr at 2024-03-29T10:57:27.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.510+0000 7fa78159a640 1 librados: starting objecter 2024-03-29T10:57:27.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.511+0000 7fa78159a640 1 Processor -- start 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.511+0000 7fa78159a640 1 -- start start 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.511+0000 7fa78159a640 1 librados: setting wanted keys 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.511+0000 7fa78159a640 1 librados: calling monclient init 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.511+0000 7fa78159a640 10 monclient: init 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 10 monclient(hunting): picked mon.noname-b con 0x7fa77c0a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 10 monclient(hunting): picked mon.noname-a con 0x7fa77c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 10 monclient(hunting): picked mon.noname-c con 0x7fa77c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c142e80 con 0x7fa77c13f650 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0720c0 con 0x7fa77c0a9d70 2024-03-29T10:57:27.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c13a4f0 con 0x7fa77c1457f0 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa78159a640 10 monclient(hunting): authenticate will time out at 1150.804199s 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa77b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa77c1457f0 0x7fa77c1e64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36488/0 (socket says 172.21.15.155:36488) 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa77b7fe640 1 -- 172.21.15.155:0/3535468782 learned_addr learned my addr 172.21.15.155:0/3535468782 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa7737fe640 10 client.?.objecter ms_handle_connect 0x7fa77c1457f0 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa7737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa7737fe640 10 client.?.objecter ms_handle_connect 0x7fa77c13f650 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.512+0000 7fa7737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311853988 0 0) 0x7fa77c142e80 con 0x7fa77c13f650 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:57:27.497 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa75c004eb0 con 0x7fa77c13f650 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826350275 0 0) 0x7fa77c13a4f0 con 0x7fa77c1457f0 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c142e80 con 0x7fa77c1457f0 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 client.?.objecter ms_handle_connect 0x7fa77c0a9d70 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1693368780 0 0) 0x7fa75c004eb0 con 0x7fa77c13f650 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa77c13a4f0 con 0x7fa77c13f650 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa76c001c70 con 0x7fa77c13f650 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:27.498 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): dump: 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 316561083 0 0) 0x7fa77c13a4f0 con 0x7fa77c13f650 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 >> v1:172.21.15.155:6790/0 conn(0x7fa77c1457f0 legacy=0x7fa77c1e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:27.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.513+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 >> v1:172.21.15.183:6789/0 conn(0x7fa77c0a9d70 legacy=0x7fa77c13b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 1 monclient: found mon.a 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c13a7d0 con 0x7fa77c13f650 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 5 monclient: authenticate success, global_id 4416 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 10 monclient: _renew_subs 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c13aab0 con 0x7fa77c13f650 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 10 monclient: _renew_subs 2024-03-29T10:57:27.500 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c139c10 con 0x7fa77c13f650 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa78159a640 1 librados: init done 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa76c003990 con 0x7fa77c13f650 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.514+0000 7fa7797fa640 4 set_mon_vals no callback set 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa76c003ca0 con 0x7fa77c13f650 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa7737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa7737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa7737fe640 10 monclient: dump: 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:27.501 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa78159a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa78159a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa78159a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:27.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.515+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa760005080 con 0x7fa77c13f650 2024-03-29T10:57:27.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.516+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fa76c019c70 con 0x7fa77c13f650 2024-03-29T10:57:27.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.517+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fa76c083c50 con 0x7fa77c13f650 2024-03-29T10:57:27.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.517+0000 7fa7737fe640 10 client.4416.objecter ms_dispatch 0x7fa77c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:27.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.517+0000 7fa77affd640 10 monclient: get_auth_request con 0x7fa75c06dd20 auth_method 0 2024-03-29T10:57:27.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.517+0000 7fa7737fe640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:27.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.517+0000 7fa7737fe640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:27.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.517+0000 7fa7737fe640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T10:57:27.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.517+0000 7fa7737fe640 10 client.4416.objecter ms_handle_connect 0x7fa75c06dd20 2024-03-29T10:57:27.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.528+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa76c04f360 con 0x7fa77c13f650 2024-03-29T10:57:27.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.528+0000 7fa7737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:27.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.528+0000 7fa7737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:27.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.903+0000 7fa78159a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:27.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.903+0000 7fa78159a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:27.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.903+0000 7fa78159a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:27.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.903+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa760005330 con 0x7fa77c13f650 2024-03-29T10:57:27.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.904+0000 7fa7737fe640 1 -- 172.21.15.155:0/3535468782 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fa76c051460 con 0x7fa77c13f650 2024-03-29T10:57:27.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.904+0000 7fa7737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:27.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.904+0000 7fa7737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:27.885 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:27.885 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:27.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.907+0000 7fa78159a640 10 librados: watch_flush enter 2024-03-29T10:57:27.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 10 librados: watch_flush exit 2024-03-29T10:57:27.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 10 client.4416.objecter successfully canceled tick 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 >> v1:172.21.15.183:6816/1412921548 conn(0x7fa75c06dd20 legacy=0x7fa75c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 10 monclient: shutdown 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 >> v1:172.21.15.155:6789/0 conn(0x7fa77c13f650 legacy=0x7fa77c140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 shutdown_connections 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 >> 172.21.15.155:0/3535468782 conn(0x7fa77c066a00 msgr2=0x7fa77c0b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 shutdown_connections 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 1 -- 172.21.15.155:0/3535468782 wait complete. 2024-03-29T10:57:27.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:27.908+0000 7fa78159a640 1 librados: shutdown 2024-03-29T10:57:27.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:34.711 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:34.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.867+0000 7f127846b640 10 monclient: get_monmap_and_config 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.867+0000 7f127846b640 10 monclient: build_initial_monmap 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.867+0000 7f127846b640 10 monclient: monmap: 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:34.868117+0000 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:34.868117+0000 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:34.848 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:34.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.868+0000 7f127846b640 1 Processor -- start 2024-03-29T10:57:34.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.868+0000 7f127846b640 1 -- start start 2024-03-29T10:57:34.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.868+0000 7f127846b640 10 monclient: init 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.869+0000 7f127846b640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 10 monclient(hunting): picked mon.noname-c con 0x7f12701457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 10 monclient(hunting): picked mon.noname-b con 0x7f127013f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1276207640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f12701457f0 0x7f127013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:54564/0 (socket says 172.21.15.155:54564) 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1276207640 1 -- 172.21.15.155:0/2619013040 learned_addr learned my addr 172.21.15.155:0/2619013040 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 10 monclient(hunting): picked mon.noname-a con 0x7f12700a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12700aaef0 con 0x7f12700a9d70 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1270145bd0 con 0x7f12701457f0 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1270145eb0 con 0x7f127013f650 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:34.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f127846b640 10 monclient(hunting): authenticate will time out at 1158.162466s 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315935536 0 0) 0x7f1270145eb0 con 0x7f127013f650 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1275205640 10 monclient(hunting): _negotiate 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1275205640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1275205640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1275205640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.870+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f125c003520 con 0x7f127013f650 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.871+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3309189643 0 0) 0x7f125c003520 con 0x7f127013f650 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.871+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 >> v1:172.21.15.183:6789/0 conn(0x7f12701457f0 legacy=0x7f127013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:34.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.871+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 >> v1:172.21.15.155:6789/0 conn(0x7f12700a9d70 legacy=0x7f127014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:34.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.872+0000 7f1275205640 10 monclient: _finish_hunting 0 2024-03-29T10:57:34.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 1 monclient: found mon.noname-b 2024-03-29T10:57:34.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:34.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:57:34.856 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1270146340 con 0x7f127013f650 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 10 monclient: _finish_auth 0 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 10 monclient: _check_auth_tickets 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f126c001690 con 0x7f127013f650 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f127846b640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f127846b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 10 monclient: dump: 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.857 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f126c001920 con 0x7f127013f650 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f127846b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.873+0000 7f1275205640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.874+0000 7f1275205640 1 -- 172.21.15.155:0/2619013040 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f126c001cb0 con 0x7f127013f650 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.874+0000 7f127846b640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.874+0000 7f127846b640 4 set_mon_vals no callback set 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.874+0000 7f1275205640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.874+0000 7f1275205640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:34.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.874+0000 7f1275205640 10 monclient: dump: 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.875+0000 7f127846b640 10 monclient: shutdown 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.875+0000 7f127846b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.875+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 >> v1:172.21.15.155:6790/0 conn(0x7f127013f650 legacy=0x7f12700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:34.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.875+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 shutdown_connections 2024-03-29T10:57:34.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.875+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 >> 172.21.15.155:0/2619013040 conn(0x7f1270066a00 msgr2=0x7f1270066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:34.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.876+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 shutdown_connections 2024-03-29T10:57:34.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.876+0000 7f127846b640 1 -- 172.21.15.155:0/2619013040 wait complete. 2024-03-29T10:57:34.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.877+0000 7f127846b640 10 monclient: build_initial_monmap 2024-03-29T10:57:34.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.877+0000 7f127846b640 10 monclient: monmap: 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.877+0000 7f127846b640 1 librados: starting msgr at 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.878+0000 7f127846b640 1 librados: starting objecter 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.878+0000 7f127846b640 1 Processor -- start 2024-03-29T10:57:34.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.878+0000 7f127846b640 1 -- start start 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.878+0000 7f127846b640 1 librados: setting wanted keys 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.878+0000 7f127846b640 1 librados: calling monclient init 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.878+0000 7f127846b640 10 monclient: init 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 10 monclient(hunting): picked mon.noname-a con 0x7f12700a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 10 monclient(hunting): picked mon.noname-c con 0x7f127013f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 10 monclient(hunting): picked mon.noname-b con 0x7f12701457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1270137ce0 con 0x7f12700a9d70 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1275a06640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f127013f650 0x7f1270140ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:49456/0 (socket says 172.21.15.155:49456) 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1275a06640 1 -- 172.21.15.155:0/3454239744 learned_addr learned my addr 172.21.15.155:0/3454239744 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12701402c0 con 0x7f12701457f0 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 client.?.objecter ms_handle_connect 0x7f127013f650 2024-03-29T10:57:34.868 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1270142c60 con 0x7f127013f650 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f127846b640 10 monclient(hunting): authenticate will time out at 1158.171302s 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 client.?.objecter ms_handle_connect 0x7f12701457f0 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683559620 0 0) 0x7f12701402c0 con 0x7f12701457f0 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1254004eb0 con 0x7f12701457f0 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041878831 0 0) 0x7f1270142c60 con 0x7f127013f650 2024-03-29T10:57:34.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.879+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12701402c0 con 0x7f127013f650 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329299854 0 0) 0x7f12701402c0 con 0x7f127013f650 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1270142c60 con 0x7f127013f650 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235947273 0 0) 0x7f1254004eb0 con 0x7f12701457f0 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12701402c0 con 0x7f12701457f0 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f126c001d30 con 0x7f127013f650 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:34.870 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1268001af0 con 0x7f12701457f0 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:34.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2508059511 0 0) 0x7f12701402c0 con 0x7f12701457f0 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 >> v1:172.21.15.155:6790/0 conn(0x7f127013f650 legacy=0x7f1270140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.881+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 >> v1:172.21.15.155:6789/0 conn(0x7f12700a9d70 legacy=0x7f12701407e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:34.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 1 monclient: found mon.b 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1270142df0 con 0x7f12701457f0 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 5 monclient: authenticate success, global_id 4423 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 10 monclient: _renew_subs 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127013fda0 con 0x7f12701457f0 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12680020c0 con 0x7f12701457f0 2024-03-29T10:57:34.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 10 monclient: _renew_subs 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1270134e30 con 0x7f12701457f0 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f127846b640 1 librados: init done 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1266ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.882+0000 7f1267fff640 4 set_mon_vals no callback set 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.883+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1268003b10 con 0x7f12701457f0 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.883+0000 7f1266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.883+0000 7f1266ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.883+0000 7f1266ffd640 10 monclient: dump: 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:34.874 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.884+0000 7f127846b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.884+0000 7f127846b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.884+0000 7f127846b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.884+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1270135fe0 con 0x7f12701457f0 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.885+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f1268019ae0 con 0x7f12701457f0 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.886+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f1268083a90 con 0x7f12701457f0 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.886+0000 7f1266ffd640 10 client.4423.objecter ms_dispatch 0x7f1270000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.886+0000 7f1266ffd640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.886+0000 7f1266ffd640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:34.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.886+0000 7f1266ffd640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T10:57:34.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.886+0000 7f1276207640 10 monclient: get_auth_request con 0x7f125406dde0 auth_method 0 2024-03-29T10:57:34.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.886+0000 7f1266ffd640 10 client.4423.objecter ms_handle_connect 0x7f125406dde0 2024-03-29T10:57:34.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.893+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f126804f1a0 con 0x7f12701457f0 2024-03-29T10:57:34.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.893+0000 7f1266ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:34.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:34.893+0000 7f1266ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:35.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.208+0000 7f127846b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:35.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.208+0000 7f127846b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:35.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.208+0000 7f127846b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:57:35.189 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.208+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1270075f10 con 0x7f12701457f0 2024-03-29T10:57:35.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.211+0000 7f1266ffd640 1 -- 172.21.15.155:0/3454239744 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f12680512a0 con 0x7f12701457f0 2024-03-29T10:57:35.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.211+0000 7f1266ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:35.191 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.211+0000 7f1266ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:35.192 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:35.192 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 10 librados: watch_flush enter 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 10 librados: watch_flush exit 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 10 client.4423.objecter successfully canceled tick 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 >> v1:172.21.15.183:6816/1412921548 conn(0x7f125406dde0 legacy=0x7f12540702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 10 monclient: shutdown 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.214+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 >> v1:172.21.15.183:6789/0 conn(0x7f12701457f0 legacy=0x7f12701e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.215+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 shutdown_connections 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.215+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 >> 172.21.15.155:0/3454239744 conn(0x7f1270066a00 msgr2=0x7f12700b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.215+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 shutdown_connections 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.215+0000 7f127846b640 1 -- 172.21.15.155:0/3454239744 wait complete. 2024-03-29T10:57:35.196 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:35.215+0000 7f127846b640 1 librados: shutdown 2024-03-29T10:57:35.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:42.019 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.163+0000 7fa8fafc8640 10 monclient: get_monmap_and_config 2024-03-29T10:57:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.163+0000 7fa8fafc8640 10 monclient: build_initial_monmap 2024-03-29T10:57:42.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.163+0000 7fa8fafc8640 10 monclient: monmap: 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:42.163814+0000 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:42.163814+0000 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:42.144 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:42.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.164+0000 7fa8fafc8640 1 Processor -- start 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.164+0000 7fa8fafc8640 1 -- start start 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.164+0000 7fa8fafc8640 10 monclient: init 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f41457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f413f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f40a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f40aaef0 con 0x7fa8f413f650 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f4145bd0 con 0x7fa8f40a9d70 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f4145eb0 con 0x7fa8f41457f0 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:42.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8fafc8640 10 monclient(hunting): authenticate will time out at 1165.457218s 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f8d64640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fa8f41457f0 0x7fa8f413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50336/0 (socket says 172.21.15.155:50336) 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f8d64640 1 -- 172.21.15.155:0/3297896606 learned_addr learned my addr 172.21.15.155:0/3297896606 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427565714 0 0) 0x7fa8f4145eb0 con 0x7fa8f41457f0 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8dc003520 con 0x7fa8f41457f0 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467606728 0 0) 0x7fa8dc003520 con 0x7fa8f41457f0 2024-03-29T10:57:42.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 >> v1:172.21.15.183:6789/0 conn(0x7fa8f40a9d70 legacy=0x7fa8f414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:42.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.165+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 >> v1:172.21.15.155:6789/0 conn(0x7fa8f413f650 legacy=0x7fa8f40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:42.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:42.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 1 monclient: found mon.noname-b 2024-03-29T10:57:42.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:42.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:57:42.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f4146340 con 0x7fa8f41457f0 2024-03-29T10:57:42.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa8e4001690 con 0x7fa8f41457f0 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8f37fe640 10 monclient: dump: 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.152 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8fafc8640 5 monclient: authenticate success, global_id 4457 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.167+0000 7fa8fafc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e40018f0 con 0x7fa8f41457f0 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8f37fe640 1 -- 172.21.15.155:0/3297896606 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa8e4001d40 con 0x7fa8f41457f0 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:42.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8f37fe640 10 monclient: dump: 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8fafc8640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:42.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.168+0000 7fa8fafc8640 4 set_mon_vals no callback set 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.169+0000 7fa8fafc8640 10 monclient: shutdown 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.169+0000 7fa8fafc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.169+0000 7fa8fafc8640 1 -- 172.21.15.155:0/3297896606 >> v1:172.21.15.155:6790/0 conn(0x7fa8f41457f0 legacy=0x7fa8f413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.169+0000 7fa8fafc8640 1 -- 172.21.15.155:0/3297896606 shutdown_connections 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.169+0000 7fa8fafc8640 1 -- 172.21.15.155:0/3297896606 >> 172.21.15.155:0/3297896606 conn(0x7fa8f4066a00 msgr2=0x7fa8f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.169+0000 7fa8fafc8640 1 -- 172.21.15.155:0/3297896606 shutdown_connections 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.169+0000 7fa8fafc8640 1 -- 172.21.15.155:0/3297896606 wait complete. 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 10 monclient: build_initial_monmap 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 10 monclient: monmap: 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:42.155 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 1 librados: starting msgr at 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 1 librados: starting objecter 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 1 Processor -- start 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 1 -- start start 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 1 librados: setting wanted keys 2024-03-29T10:57:42.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 1 librados: calling monclient init 2024-03-29T10:57:42.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.170+0000 7fa8fafc8640 10 monclient: init 2024-03-29T10:57:42.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:42.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:42.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f40a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:42.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f413f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:42.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f41457f0 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:42.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f413c570 con 0x7fa8f413f650 2024-03-29T10:57:42.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f413c810 con 0x7fa8f41457f0 2024-03-29T10:57:42.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f4138100 con 0x7fa8f40a9d70 2024-03-29T10:57:42.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.171+0000 7fa8fafc8640 10 monclient(hunting): authenticate will time out at 1165.463597s 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f9565640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fa8f41457f0 0x7fa8f41e1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:40564/0 (socket says 172.21.15.155:40564) 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f9565640 1 -- 172.21.15.155:0/406195433 learned_addr learned my addr 172.21.15.155:0/406195433 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 client.?.objecter ms_handle_connect 0x7fa8f41457f0 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 client.?.objecter ms_handle_connect 0x7fa8f40a9d70 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:42.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656284233 0 0) 0x7fa8f4138100 con 0x7fa8f40a9d70 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.172+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e0004eb0 con 0x7fa8f40a9d70 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343116268 0 0) 0x7fa8e0004eb0 con 0x7fa8f40a9d70 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8e0005300 con 0x7fa8f40a9d70 2024-03-29T10:57:42.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa8e4001650 con 0x7fa8f40a9d70 2024-03-29T10:57:42.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:42.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 10 monclient(hunting): dump: 2024-03-29T10:57:42.166 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:42.166 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:42.166 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.166 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1163909854 0 0) 0x7fa8e0005300 con 0x7fa8f40a9d70 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 >> v1:172.21.15.183:6789/0 conn(0x7fa8f41457f0 legacy=0x7fa8f41e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:42.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.173+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 >> v1:172.21.15.155:6789/0 conn(0x7fa8f413f650 legacy=0x7fa8f41d7950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.174+0000 7fa8f17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.174+0000 7fa8f17fa640 1 monclient: found mon.c 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.174+0000 7fa8f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.174+0000 7fa8f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.174+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f4138290 con 0x7fa8f40a9d70 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8f17fa640 10 monclient: _finish_auth 0 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8f17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 5 monclient: authenticate success, global_id 4460 2024-03-29T10:57:42.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 10 monclient: _renew_subs 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f4142a50 con 0x7fa8f40a9d70 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 10 monclient: _renew_subs 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f4138b50 con 0x7fa8f40a9d70 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.175+0000 7fa8fafc8640 1 librados: init done 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8fafc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:42.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8fafc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8fafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f4075f10 con 0x7fa8f40a9d70 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e4004a90 con 0x7fa8f40a9d70 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fa8e4001650 con 0x7fa8f40a9d70 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8f17fa640 10 monclient: dump: 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:42.170 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.176+0000 7fa8f2ffd640 4 set_mon_vals no callback set 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.177+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fa8e4004d00 con 0x7fa8f40a9d70 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.178+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fa8e4084830 con 0x7fa8f40a9d70 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.178+0000 7fa8f3fff640 10 monclient: get_auth_request con 0x7fa8e006e0c0 auth_method 0 2024-03-29T10:57:42.171 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.178+0000 7fa8f17fa640 10 client.4460.objecter ms_dispatch 0x7fa8f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:42.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.178+0000 7fa8f17fa640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:42.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.178+0000 7fa8f17fa640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:42.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.178+0000 7fa8f17fa640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T10:57:42.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.178+0000 7fa8f17fa640 10 client.4460.objecter ms_handle_connect 0x7fa8e006e0c0 2024-03-29T10:57:42.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.183+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8e4050110 con 0x7fa8f40a9d70 2024-03-29T10:57:42.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.183+0000 7fa8f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:42.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.183+0000 7fa8f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:42.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.468+0000 7fa8fafc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:42.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.468+0000 7fa8fafc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:42.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.468+0000 7fa8fafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:42.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.468+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8f40b4700 con 0x7fa8f40a9d70 2024-03-29T10:57:42.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.468+0000 7fa8f17fa640 1 -- 172.21.15.155:0/406195433 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fa8e4052040 con 0x7fa8f40a9d70 2024-03-29T10:57:42.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.468+0000 7fa8f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:42.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.468+0000 7fa8f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:42.450 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:42.450 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 10 librados: watch_flush enter 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 10 librados: watch_flush exit 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 10 client.4460.objecter successfully canceled tick 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 >> v1:172.21.15.183:6816/1412921548 conn(0x7fa8e006e0c0 legacy=0x7fa8e0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 10 monclient: shutdown 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:42.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 >> v1:172.21.15.155:6790/0 conn(0x7fa8f40a9d70 legacy=0x7fa8f41d7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:42.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 shutdown_connections 2024-03-29T10:57:42.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.471+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 >> 172.21.15.155:0/406195433 conn(0x7fa8f4066a00 msgr2=0x7fa8f40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:42.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.472+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 shutdown_connections 2024-03-29T10:57:42.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.472+0000 7fa8fafc8640 1 -- 172.21.15.155:0/406195433 wait complete. 2024-03-29T10:57:42.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:42.472+0000 7fa8fafc8640 1 librados: shutdown 2024-03-29T10:57:42.466 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:49.272 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:49.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.444+0000 7fbbb9dac640 10 monclient: get_monmap_and_config 2024-03-29T10:57:49.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.444+0000 7fbbb9dac640 10 monclient: build_initial_monmap 2024-03-29T10:57:49.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.445+0000 7fbbb9dac640 10 monclient: monmap: 2024-03-29T10:57:49.425 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:49.425 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:49.445676+0000 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:49.445676+0000 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:49.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.446+0000 7fbbb9dac640 1 Processor -- start 2024-03-29T10:57:49.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.446+0000 7fbbb9dac640 1 -- start start 2024-03-29T10:57:49.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.446+0000 7fbbb9dac640 10 monclient: init 2024-03-29T10:57:49.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:49.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:49.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 10 monclient(hunting): picked mon.noname-a con 0x7fbbb41422b0 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:49.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb4142690 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:49.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb40b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:49.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb4142c80 con 0x7fbbb41422b0 2024-03-29T10:57:49.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb40aae40 con 0x7fbbb40b17d0 2024-03-29T10:57:49.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb413fbd0 con 0x7fbbb4142690 2024-03-29T10:57:49.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:49.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.447+0000 7fbbb9dac640 10 monclient(hunting): authenticate will time out at 1172.739606s 2024-03-29T10:57:49.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.449+0000 7fbbb2ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fbbb4142690 0x7fbbb40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34800/0 (socket says 172.21.15.155:34800) 2024-03-29T10:57:49.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.449+0000 7fbbb2ffd640 1 -- 172.21.15.155:0/2948246691 learned_addr learned my addr 172.21.15.155:0/2948246691 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:49.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.450+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651982959 0 0) 0x7fbbb413fbd0 con 0x7fbbb4142690 2024-03-29T10:57:49.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.450+0000 7fbbb27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:49.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.450+0000 7fbbb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:49.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.450+0000 7fbbb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:49.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.450+0000 7fbbb27fc640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:57:49.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.450+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb9c003520 con 0x7fbbb4142690 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.452+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554847638 0 0) 0x7fbb9c003520 con 0x7fbbb4142690 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.452+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 >> v1:172.21.15.183:6789/0 conn(0x7fbbb40b17d0 legacy=0x7fbbb40b2ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.452+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 >> v1:172.21.15.155:6789/0 conn(0x7fbbb41422b0 legacy=0x7fbbb413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.452+0000 7fbbb27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.452+0000 7fbbb27fc640 1 monclient: found mon.noname-b 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4151940 con 0x7fbbb4142690 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:49.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbba4001690 con 0x7fbbb4142690 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb9dac640 5 monclient: authenticate success, global_id 4463 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb9dac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: dump: 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:49.436 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbba40018b0 con 0x7fbbb4142690 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb9dac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 1 -- 172.21.15.155:0/2948246691 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbba4001c40 con 0x7fbbb4142690 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.453+0000 7fbbb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.454+0000 7fbbb27fc640 10 monclient: dump: 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:49.437 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:49.438 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:49.438 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:49.438 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:49.438 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.454+0000 7fbbb9dac640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.454+0000 7fbbb9dac640 4 set_mon_vals no callback set 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.455+0000 7fbbb9dac640 10 monclient: shutdown 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.455+0000 7fbbb9dac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.455+0000 7fbbb9dac640 1 -- 172.21.15.155:0/2948246691 >> v1:172.21.15.155:6790/0 conn(0x7fbbb4142690 legacy=0x7fbbb40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.455+0000 7fbbb9dac640 1 -- 172.21.15.155:0/2948246691 shutdown_connections 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.455+0000 7fbbb9dac640 1 -- 172.21.15.155:0/2948246691 >> 172.21.15.155:0/2948246691 conn(0x7fbbb4066a00 msgr2=0x7fbbb4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.457+0000 7fbbb9dac640 1 -- 172.21.15.155:0/2948246691 shutdown_connections 2024-03-29T10:57:49.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.457+0000 7fbbb9dac640 1 -- 172.21.15.155:0/2948246691 wait complete. 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.459+0000 7fbbb9dac640 10 monclient: build_initial_monmap 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.459+0000 7fbbb9dac640 10 monclient: monmap: 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:49.440 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.459+0000 7fbbb9dac640 1 librados: starting msgr at 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.459+0000 7fbbb9dac640 1 librados: starting objecter 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.460+0000 7fbbb9dac640 1 Processor -- start 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.460+0000 7fbbb9dac640 1 -- start start 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.460+0000 7fbbb9dac640 1 librados: setting wanted keys 2024-03-29T10:57:49.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.460+0000 7fbbb9dac640 1 librados: calling monclient init 2024-03-29T10:57:49.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.460+0000 7fbbb9dac640 10 monclient: init 2024-03-29T10:57:49.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.460+0000 7fbbb9dac640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:49.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.461+0000 7fbbb9dac640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:49.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.461+0000 7fbbb9dac640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb40b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:49.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.461+0000 7fbbb9dac640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb41422b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:49.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.462+0000 7fbbb37fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fbbb40b17d0 0x7fbbb4132230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35194/0 (socket says 172.21.15.155:35194) 2024-03-29T10:57:49.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb9dac640 10 monclient(hunting): picked mon.noname-a con 0x7fbbb4142690 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:49.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb9dac640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb406ee80 con 0x7fbbb4142690 2024-03-29T10:57:49.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb9dac640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb406f160 con 0x7fbbb41422b0 2024-03-29T10:57:49.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb9dac640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb4131d10 con 0x7fbbb40b17d0 2024-03-29T10:57:49.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb9dac640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:49.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb9dac640 10 monclient(hunting): authenticate will time out at 1172.756033s 2024-03-29T10:57:49.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb3fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbbb4142690 0x7fbbb4070da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:42105/0 (socket says 172.21.15.155:42105) 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.464+0000 7fbbb3fff640 1 -- 172.21.15.155:0/1281359690 learned_addr learned my addr 172.21.15.155:0/1281359690 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 client.?.objecter ms_handle_connect 0x7fbbb4142690 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 client.?.objecter ms_handle_connect 0x7fbbb40b17d0 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057779453 0 0) 0x7fbbb406ee80 con 0x7fbbb4142690 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:49.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:57:49.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.465+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba0004eb0 con 0x7fbbb4142690 2024-03-29T10:57:49.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853478068 0 0) 0x7fbbb4131d10 con 0x7fbbb40b17d0 2024-03-29T10:57:49.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:49.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:49.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:49.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:57:49.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb406ee80 con 0x7fbbb40b17d0 2024-03-29T10:57:49.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2727327463 0 0) 0x7fbba0004eb0 con 0x7fbbb4142690 2024-03-29T10:57:49.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbb4131d10 con 0x7fbbb4142690 2024-03-29T10:57:49.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbba8001770 con 0x7fbbb4142690 2024-03-29T10:57:49.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): dump: 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:49.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 826703027 0 0) 0x7fbbb4131d10 con 0x7fbbb4142690 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 >> v1:172.21.15.155:6790/0 conn(0x7fbbb40b17d0 legacy=0x7fbbb4132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 >> v1:172.21.15.183:6789/0 conn(0x7fbbb41422b0 legacy=0x7fbbb4070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 monclient: found mon.a 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4131930 con 0x7fbbb4142690 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient: _finish_auth 0 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.467+0000 7fbb9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 5 monclient: authenticate success, global_id 4419 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 10 monclient: _renew_subs 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:49.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb406de90 con 0x7fbbb4142690 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbba80018d0 con 0x7fbbb4142690 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbb9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 10 monclient: _renew_subs 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb4132b20 con 0x7fbbb4142690 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb9dac640 1 librados: init done 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbbb17fa640 4 set_mon_vals no callback set 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fbba8001e00 con 0x7fbbb4142690 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbb9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbb9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.468+0000 7fbb9bfff640 10 monclient: dump: 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:49.452 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.453 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:49.453 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:49.453 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:49.453 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:49.453 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:49.453 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:49.453 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:49.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.470+0000 7fbb99ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:49.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.470+0000 7fbb99ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.470+0000 7fbb99ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.470+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb4131b80 con 0x7fbbb4142690 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.470+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fbba801a910 con 0x7fbbb4142690 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.471+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fbba80849c0 con 0x7fbbb4142690 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.471+0000 7fbb9bfff640 10 client.4419.objecter ms_dispatch 0x7fbbb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.471+0000 7fbb9bfff640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.471+0000 7fbbb37fe640 10 monclient: get_auth_request con 0x7fbba006db60 auth_method 0 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.471+0000 7fbb9bfff640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.471+0000 7fbb9bfff640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T10:57:49.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.471+0000 7fbb9bfff640 10 client.4419.objecter ms_handle_connect 0x7fbba006db60 2024-03-29T10:57:49.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.478+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbba80500d0 con 0x7fbbb4142690 2024-03-29T10:57:49.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.478+0000 7fbb9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:49.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.478+0000 7fbb9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:49.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.815+0000 7fbb99ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:49.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.815+0000 7fbb99ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:49.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.815+0000 7fbb99ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:57:49.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.815+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbb4132030 con 0x7fbbb4142690 2024-03-29T10:57:49.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.817+0000 7fbb9bfff640 1 -- 172.21.15.155:0/1281359690 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fbba80521d0 con 0x7fbbb4142690 2024-03-29T10:57:49.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.817+0000 7fbb9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:49.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.817+0000 7fbb9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:49.798 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:49.798 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.821+0000 7fbb99ffb640 10 librados: watch_flush enter 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.821+0000 7fbb99ffb640 10 librados: watch_flush exit 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.821+0000 7fbb99ffb640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.821+0000 7fbb99ffb640 10 client.4419.objecter successfully canceled tick 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.821+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 >> v1:172.21.15.183:6816/1412921548 conn(0x7fbba006db60 legacy=0x7fbba0070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.821+0000 7fbb99ffb640 10 monclient: shutdown 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.821+0000 7fbb99ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.822+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 >> v1:172.21.15.155:6789/0 conn(0x7fbbb4142690 legacy=0x7fbbb4070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:49.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.822+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 shutdown_connections 2024-03-29T10:57:49.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.822+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 >> 172.21.15.155:0/1281359690 conn(0x7fbbb4066a00 msgr2=0x7fbbb40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:49.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.823+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 shutdown_connections 2024-03-29T10:57:49.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.823+0000 7fbb99ffb640 1 -- 172.21.15.155:0/1281359690 wait complete. 2024-03-29T10:57:49.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:49.823+0000 7fbb99ffb640 1 librados: shutdown 2024-03-29T10:57:49.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:56.628 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:56.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.799+0000 7f5e8949a640 10 monclient: get_monmap_and_config 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.799+0000 7f5e8949a640 10 monclient: build_initial_monmap 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.799+0000 7f5e8949a640 10 monclient: monmap: 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:57:56.801261+0000 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:57:56.801261+0000 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:56.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.800+0000 7f5e8949a640 1 Processor -- start 2024-03-29T10:57:56.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.800+0000 7f5e8949a640 1 -- start start 2024-03-29T10:57:56.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.800+0000 7f5e8949a640 10 monclient: init 2024-03-29T10:57:56.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.801+0000 7f5e8949a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:56.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.801+0000 7f5e8949a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:56.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.802+0000 7f5e8949a640 10 monclient(hunting): picked mon.noname-b con 0x7f5e840a9ac0 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:56.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.802+0000 7f5e82ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5e840a9ac0 0x7f5e840adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53900/0 (socket says 172.21.15.155:53900) 2024-03-29T10:57:56.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.802+0000 7f5e82ffd640 1 -- 172.21.15.155:0/3849356686 learned_addr learned my addr 172.21.15.155:0/3849356686 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:56.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.802+0000 7f5e8949a640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8413ee90 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.803+0000 7f5e8949a640 10 monclient(hunting): picked mon.noname-c con 0x7f5e840af900 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.803+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e840afd10 con 0x7f5e8413ee90 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.803+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e840af6f0 con 0x7f5e840af900 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.803+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e840aac80 con 0x7f5e840a9ac0 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.803+0000 7f5e8949a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.803+0000 7f5e8949a640 10 monclient(hunting): authenticate will time out at 1180.096491s 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.804+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042209533 0 0) 0x7f5e840aac80 con 0x7f5e840a9ac0 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.804+0000 7f5e81ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.804+0000 7f5e81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.804+0000 7f5e81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:56.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.804+0000 7f5e81ffb640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:57:56.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.804+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e6c003520 con 0x7f5e840a9ac0 2024-03-29T10:57:56.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.805+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930845824 0 0) 0x7f5e6c003520 con 0x7f5e840a9ac0 2024-03-29T10:57:56.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.805+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 >> v1:172.21.15.183:6789/0 conn(0x7f5e840af900 legacy=0x7f5e84149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:56.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.805+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 >> v1:172.21.15.155:6789/0 conn(0x7f5e8413ee90 legacy=0x7f5e8413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:56.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:56.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 1 monclient: found mon.noname-b 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840aaf60 con 0x7f5e840a9ac0 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e8949a640 5 monclient: authenticate success, global_id 4469 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e8949a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.806+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e7c001a10 con 0x7f5e840a9ac0 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.807+0000 7f5e81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.807+0000 7f5e81ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.807+0000 7f5e81ffb640 10 monclient: dump: 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:56.789 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:56.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.807+0000 7f5e8949a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e7c001ca0 con 0x7f5e840a9ac0 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e81ffb640 1 -- 172.21.15.155:0/3849356686 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e7c002030 con 0x7f5e840a9ac0 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e81ffb640 10 monclient: dump: 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.791 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:56.792 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:56.792 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:56.792 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:56.792 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:56.792 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:56.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e8949a640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:56.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.809+0000 7f5e8949a640 4 set_mon_vals no callback set 2024-03-29T10:57:56.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.810+0000 7f5e8949a640 10 monclient: shutdown 2024-03-29T10:57:56.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.810+0000 7f5e8949a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:56.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.810+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 >> v1:172.21.15.155:6790/0 conn(0x7f5e840a9ac0 legacy=0x7f5e840adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:56.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.811+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 shutdown_connections 2024-03-29T10:57:56.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.811+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 >> 172.21.15.155:0/3849356686 conn(0x7f5e840b35e0 msgr2=0x7f5e840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:56.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.812+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 shutdown_connections 2024-03-29T10:57:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.812+0000 7f5e8949a640 1 -- 172.21.15.155:0/3849356686 wait complete. 2024-03-29T10:57:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.813+0000 7f5e8949a640 10 monclient: build_initial_monmap 2024-03-29T10:57:56.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.813+0000 7f5e8949a640 10 monclient: monmap: 2024-03-29T10:57:56.795 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.813+0000 7f5e8949a640 1 librados: starting msgr at 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.813+0000 7f5e8949a640 1 librados: starting objecter 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.813+0000 7f5e8949a640 1 Processor -- start 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.814+0000 7f5e8949a640 1 -- start start 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.814+0000 7f5e8949a640 1 librados: setting wanted keys 2024-03-29T10:57:56.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.814+0000 7f5e8949a640 1 librados: calling monclient init 2024-03-29T10:57:56.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.814+0000 7f5e8949a640 10 monclient: init 2024-03-29T10:57:56.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.815+0000 7f5e8949a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:56.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.815+0000 7f5e8949a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:56.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 10 monclient(hunting): picked mon.noname-a con 0x7f5e840a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T10:57:56.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 10 monclient(hunting): picked mon.noname-b con 0x7f5e840af900 addr v1:172.21.15.183:6789/0 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8413ee90 addr v1:172.21.15.155:6790/0 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e841517b0 con 0x7f5e840a9ac0 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e841415d0 con 0x7f5e840af900 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8412f030 con 0x7f5e8413ee90 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e8949a640 10 monclient(hunting): authenticate will time out at 1180.111188s 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e82ffd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5e840a9ac0 0x7f5e84141d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36831/0 (socket says 172.21.15.155:36831) 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e82ffd640 1 -- 172.21.15.155:0/2192744195 learned_addr learned my addr 172.21.15.155:0/2192744195 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 client.?.objecter ms_handle_connect 0x7f5e840a9ac0 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:56.801 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 client.?.objecter ms_handle_connect 0x7f5e8413ee90 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419962759 0 0) 0x7f5e841517b0 con 0x7f5e840a9ac0 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e70004eb0 con 0x7f5e840a9ac0 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978502564 0 0) 0x7f5e8412f030 con 0x7f5e8413ee90 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:57:56.802 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.818+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e841517b0 con 0x7f5e8413ee90 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2939553063 0 0) 0x7f5e841517b0 con 0x7f5e8413ee90 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e8412f030 con 0x7f5e8413ee90 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 242906558 0 0) 0x7f5e70004eb0 con 0x7f5e840a9ac0 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e841517b0 con 0x7f5e840a9ac0 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e780016e0 con 0x7f5e8413ee90 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:56.803 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e7c0018d0 con 0x7f5e840a9ac0 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 10 monclient(hunting): dump: 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:56.804 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:56.805 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:56.805 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1399906352 0 0) 0x7f5e8412f030 con 0x7f5e8413ee90 2024-03-29T10:57:56.805 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.819+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 >> v1:172.21.15.183:6789/0 conn(0x7f5e840af900 legacy=0x7f5e84150990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.822+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 >> v1:172.21.15.155:6789/0 conn(0x7f5e840a9ac0 legacy=0x7f5e84141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:56.807 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 1 monclient: found mon.c 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8412ec50 con 0x7f5e8413ee90 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e8949a640 5 monclient: authenticate success, global_id 4472 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e8949a640 10 monclient: _renew_subs 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e8949a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e84142440 con 0x7f5e8413ee90 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.823+0000 7f5e8949a640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e8949a640 10 monclient: _renew_subs 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e8949a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:56.808 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e841e74a0 con 0x7f5e8413ee90 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e8949a640 1 librados: init done 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e78001840 con 0x7f5e8413ee90 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5e78001d70 con 0x7f5e8413ee90 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.824+0000 7f5e817fa640 4 set_mon_vals no callback set 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.826+0000 7f5e6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.826+0000 7f5e6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.826+0000 7f5e6b7fe640 10 monclient: dump: 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:57:56.809 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:57:56.810 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:57:56.810 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.826+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f5e78003790 con 0x7f5e8413ee90 2024-03-29T10:57:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e82ffd640 10 monclient: get_auth_request con 0x7f5e7006de00 auth_method 0 2024-03-29T10:57:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f5e78083690 con 0x7f5e8413ee90 2024-03-29T10:57:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e6b7fe640 10 client.4472.objecter ms_dispatch 0x7f5e84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e6b7fe640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:56.811 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e6b7fe640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e6b7fe640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T10:57:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e6b7fe640 10 client.4472.objecter ms_handle_connect 0x7f5e7006de00 2024-03-29T10:57:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.827+0000 7f5e8949a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.828+0000 7f5e8949a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.828+0000 7f5e8949a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:56.812 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.828+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e64005080 con 0x7f5e8413ee90 2024-03-29T10:57:56.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.836+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e7804eda0 con 0x7f5e8413ee90 2024-03-29T10:57:56.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.836+0000 7f5e6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:56.817 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:56.836+0000 7f5e6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:57.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.161+0000 7f5e8949a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:57.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.161+0000 7f5e8949a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:57.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.161+0000 7f5e8949a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:57:57.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.161+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e64005330 con 0x7f5e8413ee90 2024-03-29T10:57:57.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.162+0000 7f5e6b7fe640 1 -- 172.21.15.155:0/2192744195 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f5e78050ea0 con 0x7f5e8413ee90 2024-03-29T10:57:57.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.162+0000 7f5e6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:57.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.162+0000 7f5e6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:57.146 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:57:57.146 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:57.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.168+0000 7f5e8949a640 10 librados: watch_flush enter 2024-03-29T10:57:57.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.169+0000 7f5e8949a640 10 librados: watch_flush exit 2024-03-29T10:57:57.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.169+0000 7f5e8949a640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T10:57:57.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.169+0000 7f5e8949a640 10 client.4472.objecter successfully canceled tick 2024-03-29T10:57:57.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.170+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 >> v1:172.21.15.183:6816/1412921548 conn(0x7f5e7006de00 legacy=0x7f5e700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.170+0000 7f5e8949a640 10 monclient: shutdown 2024-03-29T10:57:57.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.170+0000 7f5e8949a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:57.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.170+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 >> v1:172.21.15.155:6790/0 conn(0x7f5e8413ee90 legacy=0x7f5e841510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:57.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.171+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 shutdown_connections 2024-03-29T10:57:57.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.171+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 >> 172.21.15.155:0/2192744195 conn(0x7f5e840b35e0 msgr2=0x7f5e84131630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:57.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.171+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 shutdown_connections 2024-03-29T10:57:57.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.171+0000 7f5e8949a640 1 -- 172.21.15.155:0/2192744195 wait complete. 2024-03-29T10:57:57.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:57:57.171+0000 7f5e8949a640 1 librados: shutdown 2024-03-29T10:57:57.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:03.979 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:04.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.153+0000 7f7eef816640 10 monclient: get_monmap_and_config 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.153+0000 7f7eef816640 10 monclient: build_initial_monmap 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.153+0000 7f7eef816640 10 monclient: monmap: 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:04.155008+0000 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:04.155008+0000 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:04.135 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.154+0000 7f7eef816640 1 Processor -- start 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.154+0000 7f7eef816640 1 -- start start 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.154+0000 7f7eef816640 10 monclient: init 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee81422b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee8142690 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee80b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee8142c80 con 0x7f7ee80b17d0 2024-03-29T10:58:04.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80aae40 con 0x7f7ee81422b0 2024-03-29T10:58:04.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee813fbd0 con 0x7f7ee8142690 2024-03-29T10:58:04.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:04.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.155+0000 7f7eef816640 10 monclient(hunting): authenticate will time out at 1187.448277s 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7eecdb1640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f7ee8142690 0x7f7ee80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44797/0 (socket says 172.21.15.155:44797) 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7eecdb1640 1 -- 172.21.15.155:0/1862200555 learned_addr learned my addr 172.21.15.155:0/1862200555 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712758452 0 0) 0x7f7ee813fbd0 con 0x7f7ee8142690 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7edffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7edffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7edffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7edffff640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:58:04.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.162+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed4003520 con 0x7f7ee8142690 2024-03-29T10:58:04.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978983623 0 0) 0x7f7ee80aae40 con 0x7f7ee81422b0 2024-03-29T10:58:04.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:04.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:04.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:04.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:58:04.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee813fbd0 con 0x7f7ee81422b0 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1536042133 0 0) 0x7f7ed4003520 con 0x7f7ee8142690 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 >> v1:172.21.15.183:6789/0 conn(0x7f7ee81422b0 legacy=0x7f7ee813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 >> v1:172.21.15.155:6789/0 conn(0x7f7ee80b17d0 legacy=0x7f7ee80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 monclient: found mon.noname-b 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee8151940 con 0x7f7ee8142690 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7ee0001990 con 0x7f7ee8142690 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: dump: 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ee0001bb0 con 0x7f7ee8142690 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 1 -- 172.21.15.155:0/1862200555 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7ee0001f80 con 0x7f7ee8142690 2024-03-29T10:58:04.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7edffff640 10 monclient: dump: 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7eef816640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:58:04.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7eef816640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:04.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.165+0000 7f7eef816640 4 set_mon_vals no callback set 2024-03-29T10:58:04.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.166+0000 7f7eef816640 10 monclient: shutdown 2024-03-29T10:58:04.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.166+0000 7f7eef816640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:04.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.167+0000 7f7eef816640 1 -- 172.21.15.155:0/1862200555 >> v1:172.21.15.155:6790/0 conn(0x7f7ee8142690 legacy=0x7f7ee80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.167+0000 7f7eef816640 1 -- 172.21.15.155:0/1862200555 shutdown_connections 2024-03-29T10:58:04.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.167+0000 7f7eef816640 1 -- 172.21.15.155:0/1862200555 >> 172.21.15.155:0/1862200555 conn(0x7f7ee8066a00 msgr2=0x7f7ee8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:04.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.168+0000 7f7eef816640 1 -- 172.21.15.155:0/1862200555 shutdown_connections 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.168+0000 7f7eef816640 1 -- 172.21.15.155:0/1862200555 wait complete. 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.169+0000 7f7eef816640 10 monclient: build_initial_monmap 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.169+0000 7f7eef816640 10 monclient: monmap: 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:04.155 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.169+0000 7f7eef816640 1 librados: starting msgr at 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.169+0000 7f7eef816640 1 librados: starting objecter 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.169+0000 7f7eef816640 1 Processor -- start 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 1 -- start start 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 1 librados: setting wanted keys 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 1 librados: calling monclient init 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient: init 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:04.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee80b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee81422b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee8142690 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee806ffd0 con 0x7f7ee80b17d0 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80b5310 con 0x7f7ee8142690 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80b5f40 con 0x7f7ee81422b0 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.170+0000 7f7eef816640 10 monclient(hunting): authenticate will time out at 1187.463691s 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.174+0000 7f7eed5b2640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7ee80b17d0 0x7f7ee8070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:37467/0 (socket says 172.21.15.155:37467) 2024-03-29T10:58:04.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.174+0000 7f7eed5b2640 1 -- 172.21.15.155:0/2865427541 learned_addr learned my addr 172.21.15.155:0/2865427541 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.174+0000 7f7eddffb640 10 client.?.objecter ms_handle_connect 0x7f7ee80b17d0 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.175+0000 7f7eddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993525632 0 0) 0x7f7ee806ffd0 con 0x7f7ee80b17d0 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ecc004eb0 con 0x7f7ee80b17d0 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 10 client.?.objecter ms_handle_connect 0x7f7ee81422b0 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.176+0000 7f7eddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972907905 0 0) 0x7f7ee80b5f40 con 0x7f7ee81422b0 2024-03-29T10:58:04.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee806ffd0 con 0x7f7ee81422b0 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 537042362 0 0) 0x7f7ee806ffd0 con 0x7f7ee81422b0 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ee80b5f40 con 0x7f7ee81422b0 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7ee0001ec0 con 0x7f7ee81422b0 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.177+0000 7f7eddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.178+0000 7f7eddffb640 10 monclient(hunting): dump: 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.163 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.178+0000 7f7eddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.178+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 718946443 0 0) 0x7f7ee80b5f40 con 0x7f7ee81422b0 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.178+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 >> v1:172.21.15.183:6789/0 conn(0x7f7ee8142690 legacy=0x7f7ee80b5830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.178+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 >> v1:172.21.15.155:6789/0 conn(0x7f7ee80b17d0 legacy=0x7f7ee8070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.178+0000 7f7eddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.178+0000 7f7eddffb640 1 monclient: found mon.c 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:04.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee80b60d0 con 0x7f7ee81422b0 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 5 monclient: authenticate success, global_id 4478 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 10 monclient: _renew_subs 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee80b6320 con 0x7f7ee81422b0 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ee0002020 con 0x7f7ee81422b0 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7ee0004fb0 con 0x7f7ee81422b0 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 10 monclient: _renew_subs 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:04.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee8083430 con 0x7f7ee81422b0 2024-03-29T10:58:04.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7eef816640 1 librados: init done 2024-03-29T10:58:04.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.179+0000 7f7edf7fe640 4 set_mon_vals no callback set 2024-03-29T10:58:04.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.181+0000 7f7eddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:04.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.181+0000 7f7eddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:04.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.181+0000 7f7eddffb640 10 monclient: dump: 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.181+0000 7f7eef816640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.181+0000 7f7eef816640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.181+0000 7f7eef816640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.181+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee80b54a0 con 0x7f7ee81422b0 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.182+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7ee001af80 con 0x7f7ee81422b0 2024-03-29T10:58:04.168 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.182+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f7ee0084df0 con 0x7f7ee81422b0 2024-03-29T10:58:04.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.183+0000 7f7eddffb640 10 client.4478.objecter ms_dispatch 0x7f7ee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:04.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.183+0000 7f7eddffb640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:04.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.183+0000 7f7eddffb640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:04.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.183+0000 7f7eddffb640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T10:58:04.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.183+0000 7f7eed5b2640 10 monclient: get_auth_request con 0x7f7ecc06dd70 auth_method 0 2024-03-29T10:58:04.169 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.186+0000 7f7eddffb640 10 client.4478.objecter ms_handle_connect 0x7f7ecc06dd70 2024-03-29T10:58:04.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.198+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ee00506d0 con 0x7f7ee81422b0 2024-03-29T10:58:04.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.198+0000 7f7eddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:04.179 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.198+0000 7f7eddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:04.501 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.520+0000 7f7eef816640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:04.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.520+0000 7f7eef816640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:04.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.520+0000 7f7eef816640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:04.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.520+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ee8075f10 con 0x7f7ee81422b0 2024-03-29T10:58:04.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.521+0000 7f7eddffb640 1 -- 172.21.15.155:0/2865427541 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f7ee0052600 con 0x7f7ee81422b0 2024-03-29T10:58:04.502 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.521+0000 7f7eddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:04.503 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.521+0000 7f7eddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:04.503 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:04.503 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:04.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.525+0000 7f7eef816640 10 librados: watch_flush enter 2024-03-29T10:58:04.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.525+0000 7f7eef816640 10 librados: watch_flush exit 2024-03-29T10:58:04.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.525+0000 7f7eef816640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T10:58:04.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.525+0000 7f7eef816640 10 client.4478.objecter successfully canceled tick 2024-03-29T10:58:04.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 >> v1:172.21.15.183:6816/1412921548 conn(0x7f7ecc06dd70 legacy=0x7f7ecc070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 10 monclient: shutdown 2024-03-29T10:58:04.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:04.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 >> v1:172.21.15.155:6790/0 conn(0x7f7ee81422b0 legacy=0x7f7ee8071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:04.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 shutdown_connections 2024-03-29T10:58:04.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 >> 172.21.15.155:0/2865427541 conn(0x7f7ee8066a00 msgr2=0x7f7ee813dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:04.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 shutdown_connections 2024-03-29T10:58:04.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 1 -- 172.21.15.155:0/2865427541 wait complete. 2024-03-29T10:58:04.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:04.526+0000 7f7eef816640 1 librados: shutdown 2024-03-29T10:58:04.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:11.336 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:11.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.500+0000 7f39981f4640 10 monclient: get_monmap_and_config 2024-03-29T10:58:11.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.500+0000 7f39981f4640 10 monclient: build_initial_monmap 2024-03-29T10:58:11.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.500+0000 7f39981f4640 10 monclient: monmap: 2024-03-29T10:58:11.482 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:11.482 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:11.482 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:11.502399+0000 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:11.502399+0000 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:11.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.501+0000 7f39981f4640 1 Processor -- start 2024-03-29T10:58:11.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.501+0000 7f39981f4640 1 -- start start 2024-03-29T10:58:11.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.501+0000 7f39981f4640 10 monclient: init 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.502+0000 7f39981f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.502+0000 7f39981f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.502+0000 7f39981f4640 10 monclient(hunting): picked mon.noname-b con 0x7f39900b0860 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.502+0000 7f39981f4640 10 monclient(hunting): picked mon.noname-a con 0x7f3990072450 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f39981f4640 10 monclient(hunting): picked mon.noname-c con 0x7f399013ee40 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f39981f4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900ad840 con 0x7f3990072450 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f39981f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399013c830 con 0x7f399013ee40 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f39981f4640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399013c9c0 con 0x7f39900b0860 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f39981f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f39981f4640 10 monclient(hunting): authenticate will time out at 1194.796112s 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f3995f90640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f39900b0860 0x7f39900acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33712/0 (socket says 172.21.15.155:33712) 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f3995f90640 1 -- 172.21.15.155:0/106340303 learned_addr learned my addr 172.21.15.155:0/106340303 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:11.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f399578f640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3990072450 0x7f399013c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:44368/0 (socket says 172.21.15.155:44368) 2024-03-29T10:58:11.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.503+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10667799 0 0) 0x7f39900ad840 con 0x7f3990072450 2024-03-29T10:58:11.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): _negotiate 2024-03-29T10:58:11.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:11.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:11.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:58:11.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c003520 con 0x7f3990072450 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789868693 0 0) 0x7f399013c9c0 con 0x7f39900b0860 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): _negotiate 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900ad840 con 0x7f39900b0860 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516419933 0 0) 0x7f397c003520 con 0x7f3990072450 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 >> v1:172.21.15.155:6790/0 conn(0x7f39900b0860 legacy=0x7f39900acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 >> v1:172.21.15.183:6789/0 conn(0x7f399013ee40 legacy=0x7f399013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient: _finish_hunting 0 2024-03-29T10:58:11.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 monclient: found mon.noname-a 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399013f970 con 0x7f3990072450 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient: _finish_auth 0 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient: _check_auth_tickets 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f398c001690 con 0x7f3990072450 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f3994f8e640 10 monclient: dump: 2024-03-29T10:58:11.491 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:11.492 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f39981f4640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.504+0000 7f39981f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f398c0017f0 con 0x7f3990072450 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f3994f8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f3994f8e640 1 -- 172.21.15.155:0/106340303 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f398c001c00 con 0x7f3990072450 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f3994f8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f3994f8e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:11.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f3994f8e640 10 monclient: dump: 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f39981f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:11.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f39981f4640 4 set_mon_vals no callback set 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.505+0000 7f39981f4640 10 monclient: shutdown 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.506+0000 7f39981f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.506+0000 7f39981f4640 1 -- 172.21.15.155:0/106340303 >> v1:172.21.15.155:6789/0 conn(0x7f3990072450 legacy=0x7f399013c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.506+0000 7f39981f4640 1 -- 172.21.15.155:0/106340303 shutdown_connections 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.506+0000 7f39981f4640 1 -- 172.21.15.155:0/106340303 >> 172.21.15.155:0/106340303 conn(0x7f399006b660 msgr2=0x7f399006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.506+0000 7f39981f4640 1 -- 172.21.15.155:0/106340303 shutdown_connections 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.506+0000 7f39981f4640 1 -- 172.21.15.155:0/106340303 wait complete. 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.508+0000 7f39981f4640 10 monclient: build_initial_monmap 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.508+0000 7f39981f4640 10 monclient: monmap: 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:11.495 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.508+0000 7f39981f4640 1 librados: starting msgr at 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.508+0000 7f39981f4640 1 librados: starting objecter 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.509+0000 7f39981f4640 1 Processor -- start 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.509+0000 7f39981f4640 1 -- start start 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.509+0000 7f39981f4640 1 librados: setting wanted keys 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.509+0000 7f39981f4640 1 librados: calling monclient init 2024-03-29T10:58:11.496 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.509+0000 7f39981f4640 10 monclient: init 2024-03-29T10:58:11.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:11.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:11.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 10 monclient(hunting): picked mon.noname-c con 0x7f3990072450 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:11.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f3995f90640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f3990072450 0x7f3990070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35446/0 (socket says 172.21.15.155:35446) 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f3995f90640 1 -- 172.21.15.155:0/332918249 learned_addr learned my addr 172.21.15.155:0/332918249 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 10 monclient(hunting): picked mon.noname-b con 0x7f39900b0860 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 10 monclient(hunting): picked mon.noname-a con 0x7f399013ee40 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900b3a70 con 0x7f399013ee40 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900b2c50 con 0x7f39900b0860 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900b40b0 con 0x7f3990072450 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:11.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.510+0000 7f39981f4640 10 monclient(hunting): authenticate will time out at 1194.803689s 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 client.?.objecter ms_handle_connect 0x7f39900b0860 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 client.?.objecter ms_handle_connect 0x7f3990072450 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 client.?.objecter ms_handle_connect 0x7f399013ee40 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685947612 0 0) 0x7f39900b2c50 con 0x7f39900b0860 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:11.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3974004eb0 con 0x7f39900b0860 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891802207 0 0) 0x7f39900b3a70 con 0x7f399013ee40 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900b2c50 con 0x7f399013ee40 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.511+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632747886 0 0) 0x7f39900b40b0 con 0x7f3990072450 2024-03-29T10:58:11.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900b3a70 con 0x7f3990072450 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1491013407 0 0) 0x7f3974004eb0 con 0x7f39900b0860 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39900b40b0 con 0x7f39900b0860 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f398c001c00 con 0x7f39900b0860 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:11.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): dump: 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:11.514 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2683590617 0 0) 0x7f39900b2c50 con 0x7f399013ee40 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3974004eb0 con 0x7f399013ee40 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3988001a80 con 0x7f399013ee40 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 10 monclient(hunting): dump: 2024-03-29T10:58:11.515 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:11.516 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33551432 0 0) 0x7f39900b3a70 con 0x7f3990072450 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39900b2c50 con 0x7f3990072450 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.512+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1263400211 0 0) 0x7f39900b40b0 con 0x7f39900b0860 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 >> v1:172.21.15.155:6790/0 conn(0x7f3990072450 legacy=0x7f3990070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 >> v1:172.21.15.155:6789/0 conn(0x7f399013ee40 legacy=0x7f39901dea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 1 monclient: found mon.b 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900b4240 con 0x7f39900b0860 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:11.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 5 monclient: authenticate success, global_id 4429 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 10 monclient: _renew_subs 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3990142810 con 0x7f39900b0860 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 10 monclient: _renew_subs 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39900b3450 con 0x7f39900b0860 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.513+0000 7f39981f4640 1 librados: init done 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f398c004b00 con 0x7f39900b0860 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f39867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f397bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:11.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f398c004e50 con 0x7f39900b0860 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f39867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f3987fff640 4 set_mon_vals no callback set 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f39867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f39867fc640 10 monclient: dump: 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:11.519 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.515+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f398c005070 con 0x7f39900b0860 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.516+0000 7f397bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.516+0000 7f397bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.516+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3970005080 con 0x7f39900b0860 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.516+0000 7f3995f90640 10 monclient: get_auth_request con 0x7f397406e070 auth_method 0 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.516+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f398c084c30 con 0x7f39900b0860 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.517+0000 7f39867fc640 10 client.4429.objecter ms_dispatch 0x7f3990000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.517+0000 7f39867fc640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:11.520 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.517+0000 7f39867fc640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:11.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.517+0000 7f39867fc640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T10:58:11.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.517+0000 7f39867fc640 10 client.4429.objecter ms_handle_connect 0x7f397406e070 2024-03-29T10:58:11.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.524+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f398c0506f0 con 0x7f39900b0860 2024-03-29T10:58:11.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.524+0000 7f39867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:11.521 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.524+0000 7f39867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:11.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.857+0000 7f397bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:11.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.857+0000 7f397bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:11.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.857+0000 7f397bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:11.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.857+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3970005330 con 0x7f39900b0860 2024-03-29T10:58:11.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.860+0000 7f39867fc640 1 -- 172.21.15.155:0/332918249 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f398c059550 con 0x7f39900b0860 2024-03-29T10:58:11.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.861+0000 7f39867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:11.842 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:11.842 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:11.842 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.861+0000 7f39867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:11.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.864+0000 7f397bfff640 10 librados: watch_flush enter 2024-03-29T10:58:11.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.864+0000 7f397bfff640 10 librados: watch_flush exit 2024-03-29T10:58:11.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.864+0000 7f397bfff640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T10:58:11.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.864+0000 7f397bfff640 10 client.4429.objecter successfully canceled tick 2024-03-29T10:58:11.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.865+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 >> v1:172.21.15.183:6816/1412921548 conn(0x7f397406e070 legacy=0x7f3974070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.865+0000 7f397bfff640 10 monclient: shutdown 2024-03-29T10:58:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.865+0000 7f397bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.865+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 >> v1:172.21.15.183:6789/0 conn(0x7f39900b0860 legacy=0x7f39901d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.865+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 shutdown_connections 2024-03-29T10:58:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.865+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 >> 172.21.15.155:0/332918249 conn(0x7f399006b660 msgr2=0x7f39900a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.866+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 shutdown_connections 2024-03-29T10:58:11.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.866+0000 7f397bfff640 1 -- 172.21.15.155:0/332918249 wait complete. 2024-03-29T10:58:11.848 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:11.866+0000 7f397bfff640 1 librados: shutdown 2024-03-29T10:58:11.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:18.672 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:18.839 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.858+0000 7f7a9b855640 10 monclient: get_monmap_and_config 2024-03-29T10:58:18.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.858+0000 7f7a9b855640 10 monclient: build_initial_monmap 2024-03-29T10:58:18.840 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.858+0000 7f7a9b855640 10 monclient: monmap: 2024-03-29T10:58:18.840 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:18.840 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:18.840 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:18.859494+0000 2024-03-29T10:58:18.841 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:18.859494+0000 2024-03-29T10:58:18.841 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:18.841 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:18.841 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:18.841 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:18.841 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:18.842 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.863+0000 7f7a9b855640 1 Processor -- start 2024-03-29T10:58:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.863+0000 7f7a9b855640 1 -- start start 2024-03-29T10:58:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.863+0000 7f7a9b855640 10 monclient: init 2024-03-29T10:58:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 10 monclient(hunting): picked mon.noname-c con 0x7f7a940b0860 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 10 monclient(hunting): picked mon.noname-b con 0x7f7a9413ef30 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 10 monclient(hunting): picked mon.noname-a con 0x7f7a940af8e0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a940b0c40 con 0x7f7a940af8e0 2024-03-29T10:58:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a940aed10 con 0x7f7a940b0860 2024-03-29T10:58:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a940afd50 con 0x7f7a9413ef30 2024-03-29T10:58:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.864+0000 7f7a9b855640 10 monclient(hunting): authenticate will time out at 1202.156323s 2024-03-29T10:58:18.849 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.868+0000 7f7a99df2640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7a940af8e0 0x7f7a94149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:54607/0 (socket says 172.21.15.155:54607) 2024-03-29T10:58:18.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.868+0000 7f7a99df2640 1 -- 172.21.15.155:0/2397566244 learned_addr learned my addr 172.21.15.155:0/2397566244 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:18.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057207105 0 0) 0x7f7a940b0c40 con 0x7f7a940af8e0 2024-03-29T10:58:18.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:18.850 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:18.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:18.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:58:18.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a80003520 con 0x7f7a940af8e0 2024-03-29T10:58:18.851 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187084357 0 0) 0x7f7a940aed10 con 0x7f7a940b0860 2024-03-29T10:58:18.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:18.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:18.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:18.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:58:18.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a940b0c40 con 0x7f7a940b0860 2024-03-29T10:58:18.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2830474707 0 0) 0x7f7a80003520 con 0x7f7a940af8e0 2024-03-29T10:58:18.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.869+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 >> v1:172.21.15.155:6790/0 conn(0x7f7a9413ef30 legacy=0x7f7a940af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:18.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 >> v1:172.21.15.183:6789/0 conn(0x7f7a940b0860 legacy=0x7f7a9413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:18.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:18.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 1 monclient: found mon.noname-a 2024-03-29T10:58:18.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:18.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:58:18.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a940ab1b0 con 0x7f7a940af8e0 2024-03-29T10:58:18.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 10 monclient: _finish_auth 0 2024-03-29T10:58:18.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:18.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:18.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a9b855640 5 monclient: authenticate success, global_id 4437 2024-03-29T10:58:18.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a9b855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:18.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7a8c001a10 con 0x7f7a940af8e0 2024-03-29T10:58:18.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:18.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:18.863 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a8bfff640 10 monclient: dump: 2024-03-29T10:58:18.863 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:18.864 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:18.864 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.864 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.864 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:18.864 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:18.865 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:18.865 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:18.865 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:18.865 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:18.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.871+0000 7f7a9b855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:18.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a8c001c60 con 0x7f7a940af8e0 2024-03-29T10:58:18.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:18.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a8bfff640 1 -- 172.21.15.155:0/2397566244 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7a8c001ff0 con 0x7f7a940af8e0 2024-03-29T10:58:18.866 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:18.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:18.867 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a8bfff640 10 monclient: dump: 2024-03-29T10:58:18.867 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:18.867 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:18.867 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.868 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.868 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:18.868 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:18.868 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:18.868 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:18.869 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:18.869 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:18.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a9b855640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:18.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.874+0000 7f7a9b855640 4 set_mon_vals no callback set 2024-03-29T10:58:18.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.875+0000 7f7a9b855640 10 monclient: shutdown 2024-03-29T10:58:18.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.875+0000 7f7a9b855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:18.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.875+0000 7f7a9b855640 1 -- 172.21.15.155:0/2397566244 >> v1:172.21.15.155:6789/0 conn(0x7f7a940af8e0 legacy=0x7f7a94149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:18.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.876+0000 7f7a9b855640 1 -- 172.21.15.155:0/2397566244 shutdown_connections 2024-03-29T10:58:18.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.876+0000 7f7a9b855640 1 -- 172.21.15.155:0/2397566244 >> 172.21.15.155:0/2397566244 conn(0x7f7a9406b660 msgr2=0x7f7a9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:18.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.876+0000 7f7a9b855640 1 -- 172.21.15.155:0/2397566244 shutdown_connections 2024-03-29T10:58:18.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.876+0000 7f7a9b855640 1 -- 172.21.15.155:0/2397566244 wait complete. 2024-03-29T10:58:18.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.877+0000 7f7a9b855640 10 monclient: build_initial_monmap 2024-03-29T10:58:18.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.877+0000 7f7a9b855640 10 monclient: monmap: 2024-03-29T10:58:18.871 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:18.871 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:18.872 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:18.872 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:18.872 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:18.872 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:18.872 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:18.873 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:18.873 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:18.873 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:18.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.878+0000 7f7a9b855640 1 librados: starting msgr at 2024-03-29T10:58:18.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.878+0000 7f7a9b855640 1 librados: starting objecter 2024-03-29T10:58:18.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.878+0000 7f7a9b855640 1 Processor -- start 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.878+0000 7f7a9b855640 1 -- start start 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.878+0000 7f7a9b855640 1 librados: setting wanted keys 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.878+0000 7f7a9b855640 1 librados: calling monclient init 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.878+0000 7f7a9b855640 10 monclient: init 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 10 monclient(hunting): picked mon.noname-c con 0x7f7a940af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 10 monclient(hunting): picked mon.noname-a con 0x7f7a940b0860 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 10 monclient(hunting): picked mon.noname-b con 0x7f7a9413ef30 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a941378b0 con 0x7f7a940b0860 2024-03-29T10:58:18.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a941384c0 con 0x7f7a9413ef30 2024-03-29T10:58:18.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a94137ae0 con 0x7f7a940af8e0 2024-03-29T10:58:18.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:18.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.879+0000 7f7a9b855640 10 monclient(hunting): authenticate will time out at 1202.171001s 2024-03-29T10:58:18.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.885+0000 7f7a99df2640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7a9413ef30 0x7f7a941e6140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:59459/0 (socket says 172.21.15.155:59459) 2024-03-29T10:58:18.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.885+0000 7f7a99df2640 1 -- 172.21.15.155:0/1606153960 learned_addr learned my addr 172.21.15.155:0/1606153960 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:18.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 client.?.objecter ms_handle_connect 0x7f7a940b0860 2024-03-29T10:58:18.888 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 client.?.objecter ms_handle_connect 0x7f7a9413ef30 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554765632 0 0) 0x7f7a941384c0 con 0x7f7a9413ef30 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a78004eb0 con 0x7f7a9413ef30 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2857049658 0 0) 0x7f7a78004eb0 con 0x7f7a9413ef30 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a941384c0 con 0x7f7a9413ef30 2024-03-29T10:58:18.891 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7a8c001f70 con 0x7f7a9413ef30 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): dump: 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:18.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.886+0000 7f7a89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3590434103 0 0) 0x7f7a941384c0 con 0x7f7a9413ef30 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 >> v1:172.21.15.155:6790/0 conn(0x7f7a940af8e0 legacy=0x7f7a941dee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 >> v1:172.21.15.155:6789/0 conn(0x7f7a940b0860 legacy=0x7f7a94138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 1 monclient: found mon.b 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.887+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94137dc0 con 0x7f7a9413ef30 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a8c004fb0 con 0x7f7a9413ef30 2024-03-29T10:58:18.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7a8c0020d0 con 0x7f7a9413ef30 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a89ffb640 10 monclient: dump: 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:18.894 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 5 monclient: authenticate success, global_id 4435 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 10 monclient: _renew_subs 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a941380a0 con 0x7f7a9413ef30 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 10 monclient: _renew_subs 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a941df580 con 0x7f7a9413ef30 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a9b855640 1 librados: init done 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.888+0000 7f7a8b7fe640 4 set_mon_vals no callback set 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.890+0000 7f7a9b855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.890+0000 7f7a9b855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:18.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.890+0000 7f7a9b855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.890+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a74005080 con 0x7f7a9413ef30 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.891+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7a8c005380 con 0x7f7a9413ef30 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.891+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f7a8c084fe0 con 0x7f7a9413ef30 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.891+0000 7f7a89ffb640 10 client.4435.objecter ms_dispatch 0x7f7a94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.891+0000 7f7a89ffb640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.891+0000 7f7a89ffb640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.891+0000 7f7a89ffb640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-29T10:58:18.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.891+0000 7f7a995f1640 10 monclient: get_auth_request con 0x7f7a7806dbd0 auth_method 0 2024-03-29T10:58:18.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.892+0000 7f7a89ffb640 10 client.4435.objecter ms_handle_connect 0x7f7a7806dbd0 2024-03-29T10:58:18.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.897+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a8c0508c0 con 0x7f7a9413ef30 2024-03-29T10:58:18.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.898+0000 7f7a89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:18.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:18.898+0000 7f7a89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:19.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.273+0000 7f7a9b855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:19.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.273+0000 7f7a9b855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:19.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.273+0000 7f7a9b855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:19.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.273+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a74005330 con 0x7f7a9413ef30 2024-03-29T10:58:19.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.274+0000 7f7a89ffb640 1 -- 172.21.15.155:0/1606153960 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f7a8c0527f0 con 0x7f7a9413ef30 2024-03-29T10:58:19.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.274+0000 7f7a89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:19.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.274+0000 7f7a89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:19.255 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:19.256 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:19.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 10 librados: watch_flush enter 2024-03-29T10:58:19.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 10 librados: watch_flush exit 2024-03-29T10:58:19.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-29T10:58:19.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 10 client.4435.objecter successfully canceled tick 2024-03-29T10:58:19.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 >> v1:172.21.15.183:6816/1412921548 conn(0x7f7a7806dbd0 legacy=0x7f7a78070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 10 monclient: shutdown 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 >> v1:172.21.15.183:6789/0 conn(0x7f7a9413ef30 legacy=0x7f7a941e6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 shutdown_connections 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 >> 172.21.15.155:0/1606153960 conn(0x7f7a9406b660 msgr2=0x7f7a9413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 shutdown_connections 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.278+0000 7f7a9b855640 1 -- 172.21.15.155:0/1606153960 wait complete. 2024-03-29T10:58:19.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:19.279+0000 7f7a9b855640 1 librados: shutdown 2024-03-29T10:58:19.274 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:26.082 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:26.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.272+0000 7ff2060c8640 10 monclient: get_monmap_and_config 2024-03-29T10:58:26.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.272+0000 7ff2060c8640 10 monclient: build_initial_monmap 2024-03-29T10:58:26.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.272+0000 7ff2060c8640 10 monclient: monmap: 2024-03-29T10:58:26.252 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:26.252 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:26.252 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:26.272888+0000 2024-03-29T10:58:26.252 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:26.272888+0000 2024-03-29T10:58:26.253 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:26.253 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:26.253 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:26.253 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:26.253 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:26.253 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:26.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.274+0000 7ff2060c8640 1 Processor -- start 2024-03-29T10:58:26.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.274+0000 7ff2060c8640 1 -- start start 2024-03-29T10:58:26.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.274+0000 7ff2060c8640 10 monclient: init 2024-03-29T10:58:26.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:26.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:26.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 10 monclient(hunting): picked mon.noname-a con 0x7ff200149ed0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:26.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 10 monclient(hunting): picked mon.noname-b con 0x7ff200072370 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:26.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 10 monclient(hunting): picked mon.noname-c con 0x7ff20014a2b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:26.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20014aa50 con 0x7ff200149ed0 2024-03-29T10:58:26.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20014a760 con 0x7ff20014a2b0 2024-03-29T10:58:26.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff200070190 con 0x7ff200072370 2024-03-29T10:58:26.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:26.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.275+0000 7ff2060c8640 10 monclient(hunting): authenticate will time out at 1209.567379s 2024-03-29T10:58:26.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.277+0000 7ff1feffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff200072370 0x7ff20006c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51390/0 (socket says 172.21.15.155:51390) 2024-03-29T10:58:26.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.277+0000 7ff1feffd640 1 -- 172.21.15.155:0/247660955 learned_addr learned my addr 172.21.15.155:0/247660955 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:26.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.278+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776683346 0 0) 0x7ff200070190 con 0x7ff200072370 2024-03-29T10:58:26.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.278+0000 7ff1fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.278+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.278+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.278+0000 7ff1fe7fc640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:58:26.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.278+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e8003520 con 0x7ff200072370 2024-03-29T10:58:26.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.280+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168713000 0 0) 0x7ff20014a760 con 0x7ff20014a2b0 2024-03-29T10:58:26.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.280+0000 7ff1fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.280+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.280+0000 7ff1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.280+0000 7ff1fe7fc640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:58:26.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.280+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff200070190 con 0x7ff20014a2b0 2024-03-29T10:58:26.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851262573 0 0) 0x7ff1e8003520 con 0x7ff200072370 2024-03-29T10:58:26.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 >> v1:172.21.15.183:6789/0 conn(0x7ff20014a2b0 legacy=0x7ff20006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 >> v1:172.21.15.155:6789/0 conn(0x7ff200149ed0 legacy=0x7ff200071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:26.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 monclient: found mon.noname-b 2024-03-29T10:58:26.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:26.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:58:26.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200070470 con 0x7ff200072370 2024-03-29T10:58:26.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:26.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:26.266 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:26.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff1f8001a10 con 0x7ff200072370 2024-03-29T10:58:26.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:26.267 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: dump: 2024-03-29T10:58:26.267 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:26.267 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:26.268 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.268 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.268 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.268 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:26.268 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:26.268 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:26.268 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:26.269 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:26.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff2060c8640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:58:26.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff2060c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:26.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f8001b70 con 0x7ff200072370 2024-03-29T10:58:26.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:26.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 1 -- 172.21.15.155:0/247660955 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff1f8001f00 con 0x7ff200072370 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff1fe7fc640 10 monclient: dump: 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:26.270 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:26.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff2060c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:26.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.281+0000 7ff2060c8640 4 set_mon_vals no callback set 2024-03-29T10:58:26.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.282+0000 7ff2060c8640 10 monclient: shutdown 2024-03-29T10:58:26.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.282+0000 7ff2060c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:26.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.282+0000 7ff2060c8640 1 -- 172.21.15.155:0/247660955 >> v1:172.21.15.155:6790/0 conn(0x7ff200072370 legacy=0x7ff20006c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.282+0000 7ff2060c8640 1 -- 172.21.15.155:0/247660955 shutdown_connections 2024-03-29T10:58:26.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.282+0000 7ff2060c8640 1 -- 172.21.15.155:0/247660955 >> 172.21.15.155:0/247660955 conn(0x7ff2000675b0 msgr2=0x7ff2000b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.283+0000 7ff2060c8640 1 -- 172.21.15.155:0/247660955 shutdown_connections 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.283+0000 7ff2060c8640 1 -- 172.21.15.155:0/247660955 wait complete. 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 10 monclient: build_initial_monmap 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 10 monclient: monmap: 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:26.273 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 1 librados: starting msgr at 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 1 librados: starting objecter 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 1 Processor -- start 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 1 -- start start 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 1 librados: setting wanted keys 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 1 librados: calling monclient init 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.285+0000 7ff2060c8640 10 monclient: init 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 10 monclient(hunting): picked mon.noname-b con 0x7ff200072370 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 10 monclient(hunting): picked mon.noname-a con 0x7ff200149ed0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:26.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 10 monclient(hunting): picked mon.noname-c con 0x7ff20014a2b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2000af370 con 0x7ff200149ed0 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2000adec0 con 0x7ff200072370 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff200147790 con 0x7ff20014a2b0 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.286+0000 7ff2060c8640 10 monclient(hunting): authenticate will time out at 1209.578586s 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.290+0000 7ff1fffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7ff20014a2b0 0x7ff200145990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51816/0 (socket says 172.21.15.155:51816) 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.290+0000 7ff1fffff640 1 -- 172.21.15.155:0/3290467872 learned_addr learned my addr 172.21.15.155:0/3290467872 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.290+0000 7ff1e7fff640 10 client.?.objecter ms_handle_connect 0x7ff20014a2b0 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.290+0000 7ff1e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.290+0000 7ff1e7fff640 10 client.?.objecter ms_handle_connect 0x7ff200072370 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.290+0000 7ff1e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.290+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259801939 0 0) 0x7ff200147790 con 0x7ff20014a2b0 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:58:26.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec004eb0 con 0x7ff20014a2b0 2024-03-29T10:58:26.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102533410 0 0) 0x7ff2000adec0 con 0x7ff200072370 2024-03-29T10:58:26.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff200147790 con 0x7ff200072370 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1663316314 0 0) 0x7ff200147790 con 0x7ff200072370 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2000adec0 con 0x7ff200072370 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.291+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff1f0001930 con 0x7ff200072370 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:26.280 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient(hunting): dump: 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1455403664 0 0) 0x7ff1ec004eb0 con 0x7ff20014a2b0 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff200147790 con 0x7ff20014a2b0 2024-03-29T10:58:26.281 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff1f4001910 con 0x7ff20014a2b0 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient(hunting): dump: 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:26.282 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3655797086 0 0) 0x7ff2000adec0 con 0x7ff200072370 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 >> v1:172.21.15.155:6790/0 conn(0x7ff20014a2b0 legacy=0x7ff200145990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 >> v1:172.21.15.155:6789/0 conn(0x7ff200149ed0 legacy=0x7ff200145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 monclient: found mon.b 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff200143e30 con 0x7ff200072370 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient: _finish_auth 0 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff1e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 5 monclient: authenticate success, global_id 4441 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 10 monclient: _renew_subs 2024-03-29T10:58:26.283 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff200144080 con 0x7ff200072370 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 10 monclient: _renew_subs 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2001e7710 con 0x7ff200072370 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.292+0000 7ff2060c8640 1 librados: init done 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.293+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f0003610 con 0x7ff200072370 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.293+0000 7ff1e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.293+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7ff1f0003960 con 0x7ff200072370 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.293+0000 7ff1fdffb640 4 set_mon_vals no callback set 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.294+0000 7ff1e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.294+0000 7ff1e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.294+0000 7ff1e7fff640 10 monclient: dump: 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:26.284 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.296+0000 7ff2060c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.296+0000 7ff2060c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.296+0000 7ff2060c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.296+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e0005080 con 0x7ff200072370 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.297+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7ff1f0003ac0 con 0x7ff200072370 2024-03-29T10:58:26.285 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.298+0000 7ff1feffd640 10 monclient: get_auth_request con 0x7ff1ec06de80 auth_method 0 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.298+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7ff1f0083840 con 0x7ff200072370 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.298+0000 7ff1e7fff640 10 client.4441.objecter ms_dispatch 0x7ff200000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.298+0000 7ff1e7fff640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.298+0000 7ff1e7fff640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.298+0000 7ff1e7fff640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.298+0000 7ff1e7fff640 10 client.4441.objecter ms_handle_connect 0x7ff1ec06de80 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.303+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1f004f120 con 0x7ff200072370 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.303+0000 7ff1e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:26.286 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.303+0000 7ff1e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:26.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.661+0000 7ff2060c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:26.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.661+0000 7ff2060c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:26.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.661+0000 7ff2060c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:26.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.661+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1e0005330 con 0x7ff200072370 2024-03-29T10:58:26.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.662+0000 7ff1e7fff640 1 -- 172.21.15.155:0/3290467872 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7ff1f0051050 con 0x7ff200072370 2024-03-29T10:58:26.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.662+0000 7ff1e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:26.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.662+0000 7ff1e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:26.643 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:26.643 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:26.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 10 librados: watch_flush enter 2024-03-29T10:58:26.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 10 librados: watch_flush exit 2024-03-29T10:58:26.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T10:58:26.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 10 client.4441.objecter successfully canceled tick 2024-03-29T10:58:26.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 >> v1:172.21.15.183:6816/1412921548 conn(0x7ff1ec06de80 legacy=0x7ff1ec070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 10 monclient: shutdown 2024-03-29T10:58:26.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:26.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.667+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 >> v1:172.21.15.183:6789/0 conn(0x7ff200072370 legacy=0x7ff2000afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:26.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.668+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 shutdown_connections 2024-03-29T10:58:26.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.668+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 >> 172.21.15.155:0/3290467872 conn(0x7ff2000675b0 msgr2=0x7ff2000b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:26.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.669+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 shutdown_connections 2024-03-29T10:58:26.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.669+0000 7ff2060c8640 1 -- 172.21.15.155:0/3290467872 wait complete. 2024-03-29T10:58:26.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:26.669+0000 7ff2060c8640 1 librados: shutdown 2024-03-29T10:58:26.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:33.478 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:33.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.649+0000 7f16cc440640 10 monclient: get_monmap_and_config 2024-03-29T10:58:33.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.649+0000 7f16cc440640 10 monclient: build_initial_monmap 2024-03-29T10:58:33.629 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.649+0000 7f16cc440640 10 monclient: monmap: 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:33.650136+0000 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:33.650136+0000 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:33.630 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:33.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.650+0000 7f16cc440640 1 Processor -- start 2024-03-29T10:58:33.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.650+0000 7f16cc440640 1 -- start start 2024-03-29T10:58:33.631 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.650+0000 7f16cc440640 10 monclient: init 2024-03-29T10:58:33.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:33.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:33.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 10 monclient(hunting): picked mon.noname-c con 0x7f16c41422b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:33.632 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 10 monclient(hunting): picked mon.noname-b con 0x7f16c4142690 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 10 monclient(hunting): picked mon.noname-a con 0x7f16c40b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c4142c80 con 0x7f16c40b17d0 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c40aae40 con 0x7f16c41422b0 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c413fbd0 con 0x7f16c4142690 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16cc440640 10 monclient(hunting): authenticate will time out at 1216.943726s 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16ca9dd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f16c40b17d0 0x7f16c40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:57646/0 (socket says 172.21.15.155:57646) 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.651+0000 7f16ca9dd640 1 -- 172.21.15.155:0/3042468232 learned_addr learned my addr 172.21.15.155:0/3042468232 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.652+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684355610 0 0) 0x7f16c4142c80 con 0x7f16c40b17d0 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.652+0000 7f16c91da640 10 monclient(hunting): _negotiate 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.652+0000 7f16c91da640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:33.633 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.652+0000 7f16c91da640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:33.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.652+0000 7f16c91da640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:58:33.634 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.652+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b0003520 con 0x7f16c40b17d0 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297380236 0 0) 0x7f16c40aae40 con 0x7f16c41422b0 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient(hunting): _negotiate 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c4142c80 con 0x7f16c41422b0 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946626560 0 0) 0x7f16c4142c80 con 0x7f16c41422b0 2024-03-29T10:58:33.635 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 >> v1:172.21.15.155:6790/0 conn(0x7f16c4142690 legacy=0x7f16c40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 >> v1:172.21.15.155:6789/0 conn(0x7f16c40b17d0 legacy=0x7f16c40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient: _finish_hunting 0 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 monclient: found mon.noname-c 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c4151940 con 0x7f16c41422b0 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient: _finish_auth 0 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient: _check_auth_tickets 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f16c0001670 con 0x7f16c41422b0 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.653+0000 7f16c91da640 10 monclient: dump: 2024-03-29T10:58:33.636 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16cc440640 5 monclient: authenticate success, global_id 4444 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16cc440640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16c0001890 con 0x7f16c41422b0 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16c91da640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:33.637 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16c91da640 1 -- 172.21.15.155:0/3042468232 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f16c0001ba0 con 0x7f16c41422b0 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16c91da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16c91da640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.654+0000 7f16c91da640 10 monclient: dump: 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:33.638 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.655+0000 7f16cc440640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:33.639 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.655+0000 7f16cc440640 4 set_mon_vals no callback set 2024-03-29T10:58:33.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.656+0000 7f16cc440640 10 monclient: shutdown 2024-03-29T10:58:33.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.656+0000 7f16cc440640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:33.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.656+0000 7f16cc440640 1 -- 172.21.15.155:0/3042468232 >> v1:172.21.15.183:6789/0 conn(0x7f16c41422b0 legacy=0x7f16c413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:33.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.657+0000 7f16cc440640 1 -- 172.21.15.155:0/3042468232 shutdown_connections 2024-03-29T10:58:33.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.657+0000 7f16cc440640 1 -- 172.21.15.155:0/3042468232 >> 172.21.15.155:0/3042468232 conn(0x7f16c4066a00 msgr2=0x7f16c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:33.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.657+0000 7f16cc440640 1 -- 172.21.15.155:0/3042468232 shutdown_connections 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.657+0000 7f16cc440640 1 -- 172.21.15.155:0/3042468232 wait complete. 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 10 monclient: build_initial_monmap 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 10 monclient: monmap: 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:33.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 1 librados: starting msgr at 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 1 librados: starting objecter 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 1 Processor -- start 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 1 -- start start 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 1 librados: setting wanted keys 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 1 librados: calling monclient init 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.658+0000 7f16cc440640 10 monclient: init 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 10 monclient(hunting): picked mon.noname-b con 0x7f16c40b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 10 monclient(hunting): picked mon.noname-c con 0x7f16c41422b0 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 10 monclient(hunting): picked mon.noname-a con 0x7f16c4142690 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c412da00 con 0x7f16c4142690 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c40ac030 con 0x7f16c40b17d0 2024-03-29T10:58:33.643 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c4131c60 con 0x7f16c41422b0 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.659+0000 7f16cc440640 10 monclient(hunting): authenticate will time out at 1216.951851s 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16c99db640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f16c41422b0 0x7f16c412cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39738/0 (socket says 172.21.15.155:39738) 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16c99db640 1 -- 172.21.15.155:0/852238680 learned_addr learned my addr 172.21.15.155:0/852238680 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 client.?.objecter ms_handle_connect 0x7f16c41422b0 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 client.?.objecter ms_handle_connect 0x7f16c40b17d0 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 client.?.objecter ms_handle_connect 0x7f16c4142690 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869643381 0 0) 0x7f16c40ac030 con 0x7f16c40b17d0 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:33.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a8004eb0 con 0x7f16c40b17d0 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2331273241 0 0) 0x7f16a8004eb0 con 0x7f16c40b17d0 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16c40ac030 con 0x7f16c40b17d0 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f16c0001610 con 0x7f16c40b17d0 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): dump: 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.645 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412611801 0 0) 0x7f16c4131c60 con 0x7f16c41422b0 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:58:33.646 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.660+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a8004eb0 con 0x7f16c41422b0 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 239368232 0 0) 0x7f16c40ac030 con 0x7f16c40b17d0 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 >> v1:172.21.15.155:6790/0 conn(0x7f16c41422b0 legacy=0x7f16c412cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 >> v1:172.21.15.155:6789/0 conn(0x7f16c4142690 legacy=0x7f16c412d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 1 monclient: found mon.b 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c4131930 con 0x7f16c40b17d0 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 5 monclient: authenticate success, global_id 4447 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 10 monclient: _renew_subs 2024-03-29T10:58:33.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:33.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c4130290 con 0x7f16c40b17d0 2024-03-29T10:58:33.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:33.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 10 monclient: _renew_subs 2024-03-29T10:58:33.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:33.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c4132340 con 0x7f16c40b17d0 2024-03-29T10:58:33.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16cc440640 1 librados: init done 2024-03-29T10:58:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16c00038a0 con 0x7f16c40b17d0 2024-03-29T10:58:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.661+0000 7f16c89d9640 4 set_mon_vals no callback set 2024-03-29T10:58:33.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f16c0003bb0 con 0x7f16c40b17d0 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16baffd640 10 monclient: dump: 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:33.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16cc440640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16cc440640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16cc440640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.662+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac005080 con 0x7f16c40b17d0 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.663+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f16c0019b80 con 0x7f16c40b17d0 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.664+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f16c0083a00 con 0x7f16c40b17d0 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.664+0000 7f16baffd640 10 client.4447.objecter ms_dispatch 0x7f16c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.664+0000 7f16baffd640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.664+0000 7f16baffd640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.664+0000 7f16baffd640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.664+0000 7f16c99db640 10 monclient: get_auth_request con 0x7f16a806dde0 auth_method 0 2024-03-29T10:58:33.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.664+0000 7f16baffd640 10 client.4447.objecter ms_handle_connect 0x7f16a806dde0 2024-03-29T10:58:33.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.671+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16c004f2e0 con 0x7f16c40b17d0 2024-03-29T10:58:33.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.671+0000 7f16baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:33.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:33.671+0000 7f16baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:34.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.046+0000 7f16cc440640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.046+0000 7f16cc440640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.046+0000 7f16cc440640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:58:34.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.046+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16ac005330 con 0x7f16c40b17d0 2024-03-29T10:58:34.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.047+0000 7f16baffd640 1 -- 172.21.15.155:0/852238680 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f16c0051210 con 0x7f16c40b17d0 2024-03-29T10:58:34.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.047+0000 7f16baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:34.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.047+0000 7f16baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:34.028 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:34.028 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 10 librados: watch_flush enter 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 10 librados: watch_flush exit 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 10 client.4447.objecter successfully canceled tick 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 >> v1:172.21.15.183:6816/1412921548 conn(0x7f16a806dde0 legacy=0x7f16a80702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 10 monclient: shutdown 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 >> v1:172.21.15.183:6789/0 conn(0x7f16c40b17d0 legacy=0x7f16c412fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 shutdown_connections 2024-03-29T10:58:34.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.051+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 >> 172.21.15.155:0/852238680 conn(0x7f16c4066a00 msgr2=0x7f16c413f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:34.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.052+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 shutdown_connections 2024-03-29T10:58:34.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.052+0000 7f16cc440640 1 -- 172.21.15.155:0/852238680 wait complete. 2024-03-29T10:58:34.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:34.052+0000 7f16cc440640 1 librados: shutdown 2024-03-29T10:58:34.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:40.854 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:40.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.009+0000 7f1033fff640 10 monclient: get_monmap_and_config 2024-03-29T10:58:40.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.009+0000 7f1033fff640 10 monclient: build_initial_monmap 2024-03-29T10:58:40.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.009+0000 7f1033fff640 10 monclient: monmap: 2024-03-29T10:58:40.990 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:41.010603+0000 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:41.010603+0000 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:40.991 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:40.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.010+0000 7f1033fff640 1 Processor -- start 2024-03-29T10:58:40.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.010+0000 7f1033fff640 1 -- start start 2024-03-29T10:58:40.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.010+0000 7f1033fff640 10 monclient: init 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 10 monclient(hunting): picked mon.noname-b con 0x7f102c0389e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 10 monclient(hunting): picked mon.noname-a con 0x7f102c038dc0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 10 monclient(hunting): picked mon.noname-c con 0x7f102c030660 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c0c54c0 con 0x7f102c038dc0 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c036430 con 0x7f102c030660 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c039840 con 0x7f102c0389e0 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1033fff640 10 monclient(hunting): authenticate will time out at 1224.303404s 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1032ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 0x7f102c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51554/0 (socket says 172.21.15.155:51554) 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1032ffd640 1 -- 172.21.15.155:0/2084905591 learned_addr learned my addr 172.21.15.155:0/2084905591 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:40.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1032ffd640 1 -- 172.21.15.155:0/2084905591 >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 legacy=0x7f102c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:58:40.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1032ffd640 1 -- 172.21.15.155:0/2084905591 >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 legacy=0x7f102c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:58:40.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1032ffd640 1 --1- 172.21.15.155:0/2084905591 >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 0x7f102c033520 :-1 s=OPENED pgs=652984 cs=1 l=1).handle_message read tag failed 2024-03-29T10:58:40.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1032ffd640 1 --1- 172.21.15.155:0/2084905591 >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 0x7f102c033520 :-1 s=OPENED pgs=652984 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:58:40.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.011+0000 7f1031ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.155:6790/0 2024-03-29T10:58:40.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283802178 0 0) 0x7f102c036430 con 0x7f102c030660 2024-03-29T10:58:40.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:40.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:40.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c003520 con 0x7f102c030660 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877886156 0 0) 0x7f101c003520 con 0x7f102c030660 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 legacy=0x7f102c033520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.013+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 >> v1:172.21.15.155:6789/0 conn(0x7f102c038dc0 legacy=0x7f102c0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 1 monclient: found mon.noname-c 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c035690 con 0x7f102c030660 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:40.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f103404f730 con 0x7f102c030660 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1033fff640 5 monclient: authenticate success, global_id 4450 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1031ffb640 10 monclient: dump: 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:41.000 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.014+0000 7f1033fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10340008d0 con 0x7f102c030660 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1031ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1031ffb640 1 -- 172.21.15.155:0/2084905591 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f10340758b0 con 0x7f102c030660 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1033fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1033fff640 4 set_mon_vals no callback set 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1031ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.015+0000 7f1031ffb640 10 monclient: dump: 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.001 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.016+0000 7f1033fff640 10 monclient: shutdown 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.016+0000 7f1033fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.016+0000 7f1033fff640 1 -- 172.21.15.155:0/2084905591 >> v1:172.21.15.183:6789/0 conn(0x7f102c030660 legacy=0x7f102c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.016+0000 7f1033fff640 1 -- 172.21.15.155:0/2084905591 shutdown_connections 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.016+0000 7f1033fff640 1 -- 172.21.15.155:0/2084905591 >> 172.21.15.155:0/2084905591 conn(0x7f102c005c00 msgr2=0x7f102c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.017+0000 7f1033fff640 1 -- 172.21.15.155:0/2084905591 shutdown_connections 2024-03-29T10:58:41.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.017+0000 7f1033fff640 1 -- 172.21.15.155:0/2084905591 wait complete. 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.018+0000 7f1033fff640 10 monclient: build_initial_monmap 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.018+0000 7f1033fff640 10 monclient: monmap: 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.018+0000 7f1033fff640 1 librados: starting msgr at 2024-03-29T10:58:41.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.018+0000 7f1033fff640 1 librados: starting objecter 2024-03-29T10:58:41.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.018+0000 7f1033fff640 1 Processor -- start 2024-03-29T10:58:41.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 1 -- start start 2024-03-29T10:58:41.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 1 librados: setting wanted keys 2024-03-29T10:58:41.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 1 librados: calling monclient init 2024-03-29T10:58:41.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient: init 2024-03-29T10:58:41.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:41.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:41.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient(hunting): picked mon.noname-a con 0x7f102c030660 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:41.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient(hunting): picked mon.noname-c con 0x7f102c0389e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:41.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient(hunting): picked mon.noname-b con 0x7f102c02ef90 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:41.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c02e5b0 con 0x7f102c030660 2024-03-29T10:58:41.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c02e890 con 0x7f102c02ef90 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c15c790 con 0x7f102c0389e0 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.019+0000 7f1033fff640 10 monclient(hunting): authenticate will time out at 1224.311709s 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.020+0000 7f10327fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 0x7f102c15ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:51870/0 (socket says 172.21.15.155:51870) 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.020+0000 7f10327fc640 1 -- 172.21.15.155:0/3365032151 learned_addr learned my addr 172.21.15.155:0/3365032151 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.020+0000 7f101b7fe640 10 client.?.objecter ms_handle_connect 0x7f102c0389e0 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.020+0000 7f101b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836300939 0 0) 0x7f102c15c790 con 0x7f102c0389e0 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1020004eb0 con 0x7f102c0389e0 2024-03-29T10:58:41.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439549205 0 0) 0x7f1020004eb0 con 0x7f102c0389e0 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1020005300 con 0x7f102c0389e0 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1028001920 con 0x7f102c0389e0 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): dump: 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:41.011 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.021+0000 7f101b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 client.?.objecter ms_handle_connect 0x7f102c030660 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825052478 0 0) 0x7f102c02e5b0 con 0x7f102c030660 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10200058a0 con 0x7f102c030660 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 377905217 0 0) 0x7f1020005300 con 0x7f102c0389e0 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 >> v1:172.21.15.183:6789/0 conn(0x7f102c02ef90 legacy=0x7f102c02f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 >> v1:172.21.15.155:6789/0 conn(0x7f102c030660 legacy=0x7f102c0c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:41.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 1 monclient: found mon.c 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c15d3c0 con 0x7f102c0389e0 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f101b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f1033fff640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f1033fff640 10 monclient: _renew_subs 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f1033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.022+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c02d3c0 con 0x7f102c0389e0 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f1033fff640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1028001c40 con 0x7f102c0389e0 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f1033fff640 10 monclient: _renew_subs 2024-03-29T10:58:41.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f1033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c02fd70 con 0x7f102c0389e0 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f101b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f1028001920 con 0x7f102c0389e0 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f101b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f101b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f101b7fe640 10 monclient: dump: 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:41.014 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f10317fa640 4 set_mon_vals no callback set 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.023+0000 7f1033fff640 1 librados: init done 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.025+0000 7f1033fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.025+0000 7f1033fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.025+0000 7f1033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.025+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c007b90 con 0x7f102c0389e0 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.025+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f1028003920 con 0x7f102c0389e0 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.026+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f10280848c0 con 0x7f102c0389e0 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.026+0000 7f101b7fe640 10 client.4499.objecter ms_dispatch 0x7f1034000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.026+0000 7f101b7fe640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.026+0000 7f101b7fe640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:41.015 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.026+0000 7f101b7fe640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:58:41.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.026+0000 7f1032ffd640 10 monclient: get_auth_request con 0x7f102006e270 auth_method 0 2024-03-29T10:58:41.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.027+0000 7f101b7fe640 10 client.4499.objecter ms_handle_connect 0x7f102006e270 2024-03-29T10:58:41.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.032+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10280501a0 con 0x7f102c0389e0 2024-03-29T10:58:41.016 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.033+0000 7f101b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:41.017 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.033+0000 7f101b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:41.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.341+0000 7f1033fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:41.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.341+0000 7f1033fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:41.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.341+0000 7f1033fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:41.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.341+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f102c15c920 con 0x7f102c0389e0 2024-03-29T10:58:41.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.341+0000 7f101b7fe640 1 -- 172.21.15.155:0/3365032151 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f10280520d0 con 0x7f102c0389e0 2024-03-29T10:58:41.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.341+0000 7f101b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:41.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.341+0000 7f101b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:41.322 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:41.322 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.344+0000 7f1033fff640 10 librados: watch_flush enter 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.344+0000 7f1033fff640 10 librados: watch_flush exit 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.344+0000 7f1033fff640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.344+0000 7f1033fff640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.344+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 >> v1:172.21.15.183:6816/1412921548 conn(0x7f102006e270 legacy=0x7f1020070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 10 monclient: shutdown 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 >> v1:172.21.15.155:6790/0 conn(0x7f102c0389e0 legacy=0x7f102c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 shutdown_connections 2024-03-29T10:58:41.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 >> 172.21.15.155:0/3365032151 conn(0x7f102c005c00 msgr2=0x7f102c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:41.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 shutdown_connections 2024-03-29T10:58:41.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 1 -- 172.21.15.155:0/3365032151 wait complete. 2024-03-29T10:58:41.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:41.345+0000 7f1033fff640 1 librados: shutdown 2024-03-29T10:58:41.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:48.146 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:48.297 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.316+0000 7f7d6d38d640 10 monclient: get_monmap_and_config 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.316+0000 7f7d6d38d640 10 monclient: build_initial_monmap 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.316+0000 7f7d6d38d640 10 monclient: monmap: 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:48.318253+0000 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:48.318253+0000 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:48.298 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:48.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.317+0000 7f7d6d38d640 1 Processor -- start 2024-03-29T10:58:48.300 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.317+0000 7f7d6d38d640 1 -- start start 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.317+0000 7f7d6d38d640 10 monclient: init 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d680b0860 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d68072450 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d6813ee40 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d680ad840 con 0x7f7d6813ee40 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6813c830 con 0x7f7d68072450 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6813c9c0 con 0x7f7d680b0860 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.318+0000 7f7d6d38d640 10 monclient(hunting): authenticate will time out at 1231.611442s 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.319+0000 7f7d66ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f7d680b0860 0x7f7d680acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:35670/0 (socket says 172.21.15.155:35670) 2024-03-29T10:58:48.301 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.319+0000 7f7d66ffd640 1 -- 172.21.15.155:0/3460682621 learned_addr learned my addr 172.21.15.155:0/3460682621 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.322+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179261862 0 0) 0x7f7d6813c9c0 con 0x7f7d680b0860 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.322+0000 7f7d667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.322+0000 7f7d667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.322+0000 7f7d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.322+0000 7f7d667fc640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.322+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d54005730 con 0x7f7d680b0860 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 479431151 0 0) 0x7f7d54005730 con 0x7f7d680b0860 2024-03-29T10:58:48.308 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 >> v1:172.21.15.183:6789/0 conn(0x7f7d68072450 legacy=0x7f7d6813c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 >> v1:172.21.15.155:6789/0 conn(0x7f7d6813ee40 legacy=0x7f7d6813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 1 monclient: found mon.noname-b 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6813f970 con 0x7f7d680b0860 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7d58001690 con 0x7f7d680b0860 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d667fc640 10 monclient: dump: 2024-03-29T10:58:48.309 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d6d38d640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.325+0000 7f7d6d38d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d58001860 con 0x7f7d680b0860 2024-03-29T10:58:48.310 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d667fc640 1 -- 172.21.15.155:0/3460682621 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7d58001cb0 con 0x7f7d680b0860 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d667fc640 10 monclient: dump: 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:48.311 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d6d38d640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.326+0000 7f7d6d38d640 4 set_mon_vals no callback set 2024-03-29T10:58:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.327+0000 7f7d6d38d640 10 monclient: shutdown 2024-03-29T10:58:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.327+0000 7f7d6d38d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:48.312 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.327+0000 7f7d6d38d640 1 -- 172.21.15.155:0/3460682621 >> v1:172.21.15.155:6790/0 conn(0x7f7d680b0860 legacy=0x7f7d680acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.328+0000 7f7d6d38d640 1 -- 172.21.15.155:0/3460682621 shutdown_connections 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.328+0000 7f7d6d38d640 1 -- 172.21.15.155:0/3460682621 >> 172.21.15.155:0/3460682621 conn(0x7f7d6806b660 msgr2=0x7f7d6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.328+0000 7f7d6d38d640 1 -- 172.21.15.155:0/3460682621 shutdown_connections 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.328+0000 7f7d6d38d640 1 -- 172.21.15.155:0/3460682621 wait complete. 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.329+0000 7f7d6d38d640 10 monclient: build_initial_monmap 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.329+0000 7f7d6d38d640 10 monclient: monmap: 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:48.313 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.330+0000 7f7d6d38d640 1 librados: starting msgr at 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.330+0000 7f7d6d38d640 1 librados: starting objecter 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.330+0000 7f7d6d38d640 1 Processor -- start 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.330+0000 7f7d6d38d640 1 -- start start 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.330+0000 7f7d6d38d640 1 librados: setting wanted keys 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.330+0000 7f7d6d38d640 1 librados: calling monclient init 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.330+0000 7f7d6d38d640 10 monclient: init 2024-03-29T10:58:48.314 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d68072450 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d680b0860 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d6813ee40 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d681d2b00 con 0x7f7d6813ee40 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d680b1b50 con 0x7f7d680b0860 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d680b1ce0 con 0x7f7d68072450 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:48.315 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.331+0000 7f7d6d38d640 10 monclient(hunting): authenticate will time out at 1231.624168s 2024-03-29T10:58:48.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.332+0000 7f7d66ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f7d68072450 0x7f7d680b1440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36842/0 (socket says 172.21.15.155:36842) 2024-03-29T10:58:48.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.332+0000 7f7d66ffd640 1 -- 172.21.15.155:0/1644482662 learned_addr learned my addr 172.21.15.155:0/1644482662 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:48.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.332+0000 7f7d5f7fe640 10 client.?.objecter ms_handle_connect 0x7f7d68072450 2024-03-29T10:58:48.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.332+0000 7f7d5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:48.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.334+0000 7f7d5f7fe640 10 client.?.objecter ms_handle_connect 0x7f7d6813ee40 2024-03-29T10:58:48.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.334+0000 7f7d5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:48.319 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.334+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518148132 0 0) 0x7f7d681d2b00 con 0x7f7d6813ee40 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.334+0000 7f7d5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.334+0000 7f7d5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.334+0000 7f7d5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d48004eb0 con 0x7f7d6813ee40 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892347092 0 0) 0x7f7d48004eb0 con 0x7f7d6813ee40 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d681d2b00 con 0x7f7d6813ee40 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7d60001ad0 con 0x7f7d6813ee40 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.335+0000 7f7d5f7fe640 10 monclient(hunting): dump: 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.320 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 950250157 0 0) 0x7f7d681d2b00 con 0x7f7d6813ee40 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 >> v1:172.21.15.155:6790/0 conn(0x7f7d68072450 legacy=0x7f7d680b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 >> v1:172.21.15.183:6789/0 conn(0x7f7d680b0860 legacy=0x7f7d681d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 1 monclient: found mon.a 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:58:48.321 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d681d36f0 con 0x7f7d6813ee40 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 client.?.objecter ms_handle_connect 0x7f7d680b0860 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 5 monclient: authenticate success, global_id 4452 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 10 monclient: _renew_subs 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d681d39c0 con 0x7f7d6813ee40 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 10 monclient: _renew_subs 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:58:48.322 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d680b3430 con 0x7f7d6813ee40 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d6d38d640 1 librados: init done 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d60002060 con 0x7f7d6813ee40 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7d60003b10 con 0x7f7d6813ee40 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.336+0000 7f7d5f7fe640 10 monclient: dump: 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:48.323 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:48.324 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:48.324 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.337+0000 7f7d65ffb640 4 set_mon_vals no callback set 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.339+0000 7f7d6d38d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.339+0000 7f7d6d38d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.339+0000 7f7d6d38d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.340+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c005080 con 0x7f7d6813ee40 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.342+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7d60003c70 con 0x7f7d6813ee40 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.343+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f7d60083aa0 con 0x7f7d6813ee40 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.343+0000 7f7d5f7fe640 10 client.4452.objecter ms_dispatch 0x7f7d68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.343+0000 7f7d5f7fe640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.343+0000 7f7d5f7fe640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:48.325 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.343+0000 7f7d5f7fe640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-29T10:58:48.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.343+0000 7f7d66ffd640 10 monclient: get_auth_request con 0x7f7d4806dc00 auth_method 0 2024-03-29T10:58:48.326 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.345+0000 7f7d5f7fe640 10 client.4452.objecter ms_handle_connect 0x7f7d4806dc00 2024-03-29T10:58:48.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.352+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d6004f1b0 con 0x7f7d6813ee40 2024-03-29T10:58:48.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.352+0000 7f7d5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:48.333 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.352+0000 7f7d5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:48.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.679+0000 7f7d6d38d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:48.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.679+0000 7f7d6d38d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:48.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.679+0000 7f7d6d38d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T10:58:48.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.679+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d4c005330 con 0x7f7d6813ee40 2024-03-29T10:58:48.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.680+0000 7f7d5f7fe640 1 -- 172.21.15.155:0/1644482662 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f7d600512b0 con 0x7f7d6813ee40 2024-03-29T10:58:48.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.680+0000 7f7d5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:48.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.680+0000 7f7d5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:48.663 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:48.663 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:48.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 10 librados: watch_flush enter 2024-03-29T10:58:48.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 10 librados: watch_flush exit 2024-03-29T10:58:48.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-29T10:58:48.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 10 client.4452.objecter successfully canceled tick 2024-03-29T10:58:48.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 >> v1:172.21.15.183:6816/1412921548 conn(0x7f7d4806dc00 legacy=0x7f7d480700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 10 monclient: shutdown 2024-03-29T10:58:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 >> v1:172.21.15.155:6789/0 conn(0x7f7d6813ee40 legacy=0x7f7d680b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 shutdown_connections 2024-03-29T10:58:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 >> 172.21.15.155:0/1644482662 conn(0x7f7d6806b660 msgr2=0x7f7d68069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 shutdown_connections 2024-03-29T10:58:48.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.684+0000 7f7d6d38d640 1 -- 172.21.15.155:0/1644482662 wait complete. 2024-03-29T10:58:48.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:48.686+0000 7f7d6d38d640 1 librados: shutdown 2024-03-29T10:58:48.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:55.491 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:55.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.660+0000 7fc30de86640 10 monclient: get_monmap_and_config 2024-03-29T10:58:55.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.660+0000 7fc30de86640 10 monclient: build_initial_monmap 2024-03-29T10:58:55.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.660+0000 7fc30de86640 10 monclient: monmap: 2024-03-29T10:58:55.641 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:55.641 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:55.641 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:58:55.661948+0000 2024-03-29T10:58:55.642 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:58:55.661948+0000 2024-03-29T10:58:55.642 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:55.642 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:55.642 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:55.642 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:58:55.642 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:58:55.642 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.661+0000 7fc30de86640 1 Processor -- start 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.661+0000 7fc30de86640 1 -- start start 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.661+0000 7fc30de86640 10 monclient: init 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 10 monclient(hunting): picked mon.noname-b con 0x7fc3081457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 10 monclient(hunting): picked mon.noname-a con 0x7fc30813f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 10 monclient(hunting): picked mon.noname-c con 0x7fc3080a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3080aaef0 con 0x7fc30813f650 2024-03-29T10:58:55.644 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc308145bd0 con 0x7fc3080a9d70 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc308145eb0 con 0x7fc3081457f0 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.662+0000 7fc30de86640 10 monclient(hunting): authenticate will time out at 1238.955171s 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.663+0000 7fc3077fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc3081457f0 0x7fc30813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39014/0 (socket says 172.21.15.155:39014) 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.663+0000 7fc3077fe640 1 -- 172.21.15.155:0/1627202637 learned_addr learned my addr 172.21.15.155:0/1627202637 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.664+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357500792 0 0) 0x7fc308145eb0 con 0x7fc3081457f0 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.664+0000 7fc3067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.664+0000 7fc3067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.664+0000 7fc3067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.664+0000 7fc3067fc640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:58:55.645 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.664+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0003520 con 0x7fc3081457f0 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 280353897 0 0) 0x7fc2f0003520 con 0x7fc3081457f0 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 >> v1:172.21.15.183:6789/0 conn(0x7fc3080a9d70 legacy=0x7fc30814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 >> v1:172.21.15.155:6789/0 conn(0x7fc30813f650 legacy=0x7fc3080b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 1 monclient: found mon.noname-b 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308146340 con 0x7fc3081457f0 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc30de86640 5 monclient: authenticate success, global_id 4508 2024-03-29T10:58:55.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc30de86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc2f8001690 con 0x7fc3081457f0 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc3067fc640 10 monclient: dump: 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.666+0000 7fc30de86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:55.649 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.667+0000 7fc3067fc640 1 -- 172.21.15.155:0/1627202637 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2f8001920 con 0x7fc3081457f0 2024-03-29T10:58:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.667+0000 7fc3067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.667+0000 7fc30de86640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.667+0000 7fc30de86640 4 set_mon_vals no callback set 2024-03-29T10:58:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.668+0000 7fc30de86640 10 monclient: shutdown 2024-03-29T10:58:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.668+0000 7fc30de86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:55.650 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.668+0000 7fc30de86640 1 -- 172.21.15.155:0/1627202637 >> v1:172.21.15.155:6790/0 conn(0x7fc3081457f0 legacy=0x7fc30813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:55.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.668+0000 7fc30de86640 1 -- 172.21.15.155:0/1627202637 shutdown_connections 2024-03-29T10:58:55.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.668+0000 7fc30de86640 1 -- 172.21.15.155:0/1627202637 >> 172.21.15.155:0/1627202637 conn(0x7fc308066a00 msgr2=0x7fc308066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:55.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.668+0000 7fc30de86640 1 -- 172.21.15.155:0/1627202637 shutdown_connections 2024-03-29T10:58:55.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.670+0000 7fc30de86640 1 -- 172.21.15.155:0/1627202637 wait complete. 2024-03-29T10:58:55.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.671+0000 7fc30de86640 10 monclient: build_initial_monmap 2024-03-29T10:58:55.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.671+0000 7fc30de86640 10 monclient: monmap: 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.671+0000 7fc30de86640 1 librados: starting msgr at 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.671+0000 7fc30de86640 1 librados: starting objecter 2024-03-29T10:58:55.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.671+0000 7fc30de86640 1 Processor -- start 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.672+0000 7fc30de86640 1 -- start start 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.672+0000 7fc30de86640 1 librados: setting wanted keys 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.672+0000 7fc30de86640 1 librados: calling monclient init 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.672+0000 7fc30de86640 10 monclient: init 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.672+0000 7fc30de86640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.672+0000 7fc30de86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.673+0000 7fc30de86640 10 monclient(hunting): picked mon.noname-c con 0x7fc3080a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T10:58:55.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.673+0000 7fc30de86640 10 monclient(hunting): picked mon.noname-b con 0x7fc30813f650 addr v1:172.21.15.183:6789/0 2024-03-29T10:58:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc3077fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc3080a9d70 0x7fc308143320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40652/0 (socket says 172.21.15.155:40652) 2024-03-29T10:58:55.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc307fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc3081457f0 0x7fc3081de910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:38927/0 (socket says 172.21.15.155:38927) 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc3077fe640 1 -- 172.21.15.155:0/1015572649 learned_addr learned my addr 172.21.15.155:0/1015572649 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc30de86640 10 monclient(hunting): picked mon.noname-a con 0x7fc3081457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc308144bd0 con 0x7fc3081457f0 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30813e4b0 con 0x7fc30813f650 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc308143a30 con 0x7fc3080a9d70 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.676+0000 7fc30de86640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc30de86640 10 monclient(hunting): authenticate will time out at 1238.969882s 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 client.?.objecter ms_handle_connect 0x7fc3080a9d70 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 client.?.objecter ms_handle_connect 0x7fc3081457f0 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061064213 0 0) 0x7fc308143a30 con 0x7fc3080a9d70 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:55.660 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f4004eb0 con 0x7fc3080a9d70 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380454866 0 0) 0x7fc308144bd0 con 0x7fc3081457f0 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.677+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc308143a30 con 0x7fc3081457f0 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3179156136 0 0) 0x7fc2f4004eb0 con 0x7fc3080a9d70 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc308144bd0 con 0x7fc3080a9d70 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc2f8001d70 con 0x7fc3080a9d70 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 10 monclient(hunting): dump: 2024-03-29T10:58:55.661 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1416008216 0 0) 0x7fc308144bd0 con 0x7fc3080a9d70 2024-03-29T10:58:55.662 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.678+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 >> v1:172.21.15.183:6789/0 conn(0x7fc30813f650 legacy=0x7fc30813cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.679+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 >> v1:172.21.15.155:6789/0 conn(0x7fc3081457f0 legacy=0x7fc3081de910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.679+0000 7fc2effff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.679+0000 7fc2effff640 1 monclient: found mon.c 2024-03-29T10:58:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.679+0000 7fc2effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:55.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308143bc0 con 0x7fc3080a9d70 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 10 monclient: _finish_auth 0 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2f8004df0 con 0x7fc3080a9d70 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc2f8005240 con 0x7fc3080a9d70 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.680+0000 7fc305ffb640 4 set_mon_vals no callback set 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc2effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc2effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc2effff640 10 monclient: dump: 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:58:55.666 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 10 monclient: _renew_subs 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30813c9d0 con 0x7fc3080a9d70 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 10 monclient: _renew_subs 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:55.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3081451d0 con 0x7fc3080a9d70 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.681+0000 7fc30de86640 1 librados: init done 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.683+0000 7fc30de86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.683+0000 7fc30de86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.683+0000 7fc30de86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.683+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30813e640 con 0x7fc3080a9d70 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.683+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fc2f80014f0 con 0x7fc3080a9d70 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.684+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fc2f8084ce0 con 0x7fc3080a9d70 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.684+0000 7fc2effff640 10 client.4511.objecter ms_dispatch 0x7fc308000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.684+0000 7fc2effff640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.684+0000 7fc2effff640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.684+0000 7fc306ffd640 10 monclient: get_auth_request con 0x7fc2f406db50 auth_method 0 2024-03-29T10:58:55.668 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.684+0000 7fc2effff640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T10:58:55.669 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.687+0000 7fc2effff640 10 client.4511.objecter ms_handle_connect 0x7fc2f406db50 2024-03-29T10:58:55.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.692+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2f80505c0 con 0x7fc3080a9d70 2024-03-29T10:58:55.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.692+0000 7fc2effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:55.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:55.692+0000 7fc2effff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.025+0000 7fc30de86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.025+0000 7fc30de86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.025+0000 7fc30de86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:58:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.025+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc30813cb60 con 0x7fc3080a9d70 2024-03-29T10:58:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.026+0000 7fc2effff640 1 -- 172.21.15.155:0/1015572649 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fc2f80524f0 con 0x7fc3080a9d70 2024-03-29T10:58:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.026+0000 7fc2effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:56.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.026+0000 7fc2effff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:56.008 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:58:56.008 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 10 librados: watch_flush enter 2024-03-29T10:58:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 10 librados: watch_flush exit 2024-03-29T10:58:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T10:58:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 10 client.4511.objecter successfully canceled tick 2024-03-29T10:58:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 >> v1:172.21.15.183:6816/1412921548 conn(0x7fc2f406db50 legacy=0x7fc2f4070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 10 monclient: shutdown 2024-03-29T10:58:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 >> v1:172.21.15.155:6790/0 conn(0x7fc3080a9d70 legacy=0x7fc308143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 shutdown_connections 2024-03-29T10:58:56.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.030+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 >> 172.21.15.155:0/1015572649 conn(0x7fc308066a00 msgr2=0x7fc30813a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:56.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.031+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 shutdown_connections 2024-03-29T10:58:56.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.031+0000 7fc30de86640 1 -- 172.21.15.155:0/1015572649 wait complete. 2024-03-29T10:58:56.013 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:58:56.031+0000 7fc30de86640 1 librados: shutdown 2024-03-29T10:58:56.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:02.835 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.002+0000 7f301c6cf640 10 monclient: get_monmap_and_config 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.002+0000 7f301c6cf640 10 monclient: build_initial_monmap 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.002+0000 7f301c6cf640 10 monclient: monmap: 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:03.003888+0000 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:03.003888+0000 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:02.983 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:02.984 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:02.984 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:02.984 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:02.984 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.003+0000 7f301c6cf640 1 Processor -- start 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.003+0000 7f301c6cf640 1 -- start start 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.003+0000 7f301c6cf640 10 monclient: init 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f30140b1f60 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f3014071eb0 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f3014143d10 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:02.985 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301406ef80 con 0x7f3014143d10 2024-03-29T10:59:02.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301406f260 con 0x7f3014071eb0 2024-03-29T10:59:02.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30140b40c0 con 0x7f30140b1f60 2024-03-29T10:59:02.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:02.986 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.004+0000 7f301c6cf640 10 monclient(hunting): authenticate will time out at 1246.297260s 2024-03-29T10:59:02.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.007+0000 7f301a46b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f30140b1f60 0x7f3014143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:60448/0 (socket says 172.21.15.155:60448) 2024-03-29T10:59:02.988 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.007+0000 7f301a46b640 1 -- 172.21.15.155:0/297861975 learned_addr learned my addr 172.21.15.155:0/297861975 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:02.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661683175 0 0) 0x7f30140b40c0 con 0x7f30140b1f60 2024-03-29T10:59:02.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 10 monclient(hunting): _negotiate 2024-03-29T10:59:02.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:02.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:02.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:59:02.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3000003520 con 0x7f30140b1f60 2024-03-29T10:59:02.989 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3252682984 0 0) 0x7f3000003520 con 0x7f30140b1f60 2024-03-29T10:59:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 >> v1:172.21.15.183:6789/0 conn(0x7f3014071eb0 legacy=0x7f3014146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:02.990 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.008+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 >> v1:172.21.15.155:6789/0 conn(0x7f3014143d10 legacy=0x7f3014146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 10 monclient: _finish_hunting 0 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 1 monclient: found mon.noname-b 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3014155d70 con 0x7f30140b1f60 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 10 monclient: _finish_auth 0 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 10 monclient: _check_auth_tickets 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3010001690 con 0x7f30140b1f60 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:02.992 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.009+0000 7f3019469640 10 monclient: dump: 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.010+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3010001860 con 0x7f30140b1f60 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.010+0000 7f3019469640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:02.993 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.010+0000 7f3019469640 1 -- 172.21.15.155:0/297861975 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3010001cb0 con 0x7f30140b1f60 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.010+0000 7f3019469640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.010+0000 7f3019469640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.010+0000 7f3019469640 10 monclient: dump: 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:02.994 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.011+0000 7f301c6cf640 5 monclient: authenticate success, global_id 4514 2024-03-29T10:59:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.011+0000 7f301c6cf640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.011+0000 7f301c6cf640 4 set_mon_vals no callback set 2024-03-29T10:59:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.011+0000 7f301c6cf640 10 monclient: shutdown 2024-03-29T10:59:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.011+0000 7f301c6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:02.995 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.011+0000 7f301c6cf640 1 -- 172.21.15.155:0/297861975 >> v1:172.21.15.155:6790/0 conn(0x7f30140b1f60 legacy=0x7f3014143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:02.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.013+0000 7f301c6cf640 1 -- 172.21.15.155:0/297861975 shutdown_connections 2024-03-29T10:59:02.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.013+0000 7f301c6cf640 1 -- 172.21.15.155:0/297861975 >> 172.21.15.155:0/297861975 conn(0x7f3014066a00 msgr2=0x7f3014066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:02.996 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.014+0000 7f301c6cf640 1 -- 172.21.15.155:0/297861975 shutdown_connections 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.014+0000 7f301c6cf640 1 -- 172.21.15.155:0/297861975 wait complete. 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.015+0000 7f301c6cf640 10 monclient: build_initial_monmap 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.015+0000 7f301c6cf640 10 monclient: monmap: 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:02.998 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.016+0000 7f301c6cf640 1 librados: starting msgr at 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.016+0000 7f301c6cf640 1 librados: starting objecter 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.016+0000 7f301c6cf640 1 Processor -- start 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.016+0000 7f301c6cf640 1 -- start start 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.016+0000 7f301c6cf640 1 librados: setting wanted keys 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.016+0000 7f301c6cf640 1 librados: calling monclient init 2024-03-29T10:59:02.999 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.016+0000 7f301c6cf640 10 monclient: init 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f30140b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f3014143d10 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f30140ab8a0 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30140ab490 con 0x7f3014143d10 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30140aca70 con 0x7f30140b1f60 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30140acd50 con 0x7f30140ab8a0 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:03.000 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.018+0000 7f301c6cf640 10 monclient(hunting): authenticate will time out at 1246.311028s 2024-03-29T10:59:03.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.019+0000 7f301ac6c640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f30140ab8a0 0x7f30140a9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:34262/0 (socket says 172.21.15.155:34262) 2024-03-29T10:59:03.001 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.019+0000 7f301ac6c640 1 -- 172.21.15.155:0/3793862188 learned_addr learned my addr 172.21.15.155:0/3793862188 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:03.002 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.020+0000 7f300affd640 10 client.?.objecter ms_handle_connect 0x7f30140ab8a0 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.020+0000 7f300affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.020+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330448866 0 0) 0x7f30140acd50 con 0x7f30140ab8a0 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.020+0000 7f300affd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.020+0000 7f300affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.020+0000 7f300affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff8004eb0 con 0x7f30140ab8a0 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 10 client.?.objecter ms_handle_connect 0x7f30140b1f60 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 245390502 0 0) 0x7f2ff8004eb0 con 0x7f30140ab8a0 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30140acd50 con 0x7f30140ab8a0 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f3004003f60 con 0x7f30140ab8a0 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.003 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 10 monclient(hunting): dump: 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:03.004 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.021+0000 7f300affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.024+0000 7f300affd640 10 client.?.objecter ms_handle_connect 0x7f3014143d10 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.024+0000 7f300affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.024+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179212491 0 0) 0x7f30140aca70 con 0x7f30140b1f60 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:59:03.007 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff8004eb0 con 0x7f30140b1f60 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 847723252 0 0) 0x7f30140acd50 con 0x7f30140ab8a0 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 >> v1:172.21.15.183:6789/0 conn(0x7f30140b1f60 legacy=0x7f3014070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 >> v1:172.21.15.155:6789/0 conn(0x7f3014143d10 legacy=0x7f30140a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 1 monclient: found mon.c 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3014152a00 con 0x7f30140ab8a0 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient: _finish_auth 0 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f300affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f301c6cf640 5 monclient: authenticate success, global_id 4517 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f301c6cf640 10 monclient: _renew_subs 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f301c6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:03.008 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.025+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3014200c50 con 0x7f30140ab8a0 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.026+0000 7f301c6cf640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.026+0000 7f301c6cf640 10 monclient: _renew_subs 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.026+0000 7f301c6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.026+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3014200fc0 con 0x7f30140ab8a0 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.026+0000 7f301c6cf640 1 librados: init done 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f301c6cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f301c6cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.009 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f301c6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30140ab220 con 0x7f30140ab8a0 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3004000ed0 con 0x7f30140ab8a0 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f300affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f30040040c0 con 0x7f30140ab8a0 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f300affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f300affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f300affd640 10 monclient: dump: 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:03.010 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:03.011 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:03.011 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:03.011 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:03.011 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:03.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f30040012e0 con 0x7f30140ab8a0 2024-03-29T10:59:03.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.028+0000 7f3018c68640 4 set_mon_vals no callback set 2024-03-29T10:59:03.011 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.029+0000 7f301a46b640 10 monclient: get_auth_request con 0x7f2ff806dd90 auth_method 0 2024-03-29T10:59:03.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.029+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f3004084a20 con 0x7f30140ab8a0 2024-03-29T10:59:03.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.029+0000 7f300affd640 10 client.4517.objecter ms_dispatch 0x7f3014000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:03.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.029+0000 7f300affd640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:03.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.029+0000 7f300affd640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:03.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.029+0000 7f300affd640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T10:59:03.012 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.030+0000 7f300affd640 10 client.4517.objecter ms_handle_connect 0x7f2ff806dd90 2024-03-29T10:59:03.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.038+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3004050380 con 0x7f30140ab8a0 2024-03-29T10:59:03.018 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.038+0000 7f300affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:03.019 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.038+0000 7f300affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:03.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.361+0000 7f301c6cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:03.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.361+0000 7f301c6cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:03.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.361+0000 7f301c6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:03.342 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.361+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3014152b90 con 0x7f30140ab8a0 2024-03-29T10:59:03.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.362+0000 7f300affd640 1 -- 172.21.15.155:0/3793862188 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f30040522b0 con 0x7f30140ab8a0 2024-03-29T10:59:03.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.362+0000 7f300affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:03.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.362+0000 7f300affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:03.343 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:03.343 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:03.346 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.365+0000 7f301c6cf640 10 librados: watch_flush enter 2024-03-29T10:59:03.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 10 librados: watch_flush exit 2024-03-29T10:59:03.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 10 client.4517.objecter successfully canceled tick 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 >> v1:172.21.15.183:6816/1412921548 conn(0x7f2ff806dd90 legacy=0x7f2ff8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 10 monclient: shutdown 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 >> v1:172.21.15.155:6790/0 conn(0x7f30140ab8a0 legacy=0x7f30140a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 shutdown_connections 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.366+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 >> 172.21.15.155:0/3793862188 conn(0x7f3014066a00 msgr2=0x7f30140b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.367+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 shutdown_connections 2024-03-29T10:59:03.348 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.367+0000 7f301c6cf640 1 -- 172.21.15.155:0/3793862188 wait complete. 2024-03-29T10:59:03.349 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:03.367+0000 7f301c6cf640 1 librados: shutdown 2024-03-29T10:59:03.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:10.173 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:10.329 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.349+0000 7f13affff640 10 monclient: get_monmap_and_config 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.349+0000 7f13affff640 10 monclient: build_initial_monmap 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.349+0000 7f13affff640 10 monclient: monmap: 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:10.350371+0000 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:10.350371+0000 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:10.330 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.350+0000 7f13affff640 1 Processor -- start 2024-03-29T10:59:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.350+0000 7f13affff640 1 -- start start 2024-03-29T10:59:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.350+0000 7f13affff640 10 monclient: init 2024-03-29T10:59:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:10.331 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:10.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 10 monclient(hunting): picked mon.noname-b con 0x7f13b00b2800 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:10.332 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 10 monclient(hunting): picked mon.noname-c con 0x7f13b00a9c00 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:10.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 10 monclient(hunting): picked mon.noname-a con 0x7f13b00b38c0 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:10.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0141eb0 con 0x7f13b00b38c0 2024-03-29T10:59:10.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b00b22d0 con 0x7f13b00a9c00 2024-03-29T10:59:10.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0142d30 con 0x7f13b00b2800 2024-03-29T10:59:10.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:10.334 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.351+0000 7f13affff640 10 monclient(hunting): authenticate will time out at 1253.643568s 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13af7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f13b00b38c0 0x7f13b014d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:40809/0 (socket says 172.21.15.155:40809) 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13aeffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f13b00b2800 0x7f13b0142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:50219/0 (socket says 172.21.15.155:50219) 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13aeffd640 1 -- 172.21.15.155:0/1148821131 learned_addr learned my addr 172.21.15.155:0/1148821131 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633489435 0 0) 0x7f13b0142d30 con 0x7f13b00b2800 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:10.338 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1398003520 con 0x7f13b00b2800 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3770388973 0 0) 0x7f1398003520 con 0x7f13b00b2800 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 >> v1:172.21.15.183:6789/0 conn(0x7f13b00a9c00 legacy=0x7f13b013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 >> v1:172.21.15.155:6789/0 conn(0x7f13b00b38c0 legacy=0x7f13b014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 1 monclient: found mon.noname-b 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b00aaf90 con 0x7f13b00b2800 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:10.339 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.355+0000 7f13adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13affff640 5 monclient: authenticate success, global_id 4520 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13affff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f13a4001690 con 0x7f13b00b2800 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13adffb640 10 monclient: dump: 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:10.340 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13affff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13adffb640 1 -- 172.21.15.155:0/1148821131 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13a40018b0 con 0x7f13b00b2800 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13affff640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.356+0000 7f13affff640 4 set_mon_vals no callback set 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.357+0000 7f13affff640 10 monclient: shutdown 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.357+0000 7f13affff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.357+0000 7f13affff640 1 -- 172.21.15.155:0/1148821131 >> v1:172.21.15.155:6790/0 conn(0x7f13b00b2800 legacy=0x7f13b0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.357+0000 7f13affff640 1 -- 172.21.15.155:0/1148821131 shutdown_connections 2024-03-29T10:59:10.341 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.357+0000 7f13affff640 1 -- 172.21.15.155:0/1148821131 >> 172.21.15.155:0/1148821131 conn(0x7f13b0066a00 msgr2=0x7f13b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.359+0000 7f13affff640 1 -- 172.21.15.155:0/1148821131 shutdown_connections 2024-03-29T10:59:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.360+0000 7f13affff640 1 -- 172.21.15.155:0/1148821131 wait complete. 2024-03-29T10:59:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.360+0000 7f13affff640 10 monclient: build_initial_monmap 2024-03-29T10:59:10.343 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.360+0000 7f13affff640 10 monclient: monmap: 2024-03-29T10:59:10.343 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:10.343 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.361+0000 7f13affff640 1 librados: starting msgr at 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.361+0000 7f13affff640 1 librados: starting objecter 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.361+0000 7f13affff640 1 Processor -- start 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.361+0000 7f13affff640 1 -- start start 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.361+0000 7f13affff640 1 librados: setting wanted keys 2024-03-29T10:59:10.344 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.361+0000 7f13affff640 1 librados: calling monclient init 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.361+0000 7f13affff640 10 monclient: init 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 10 monclient(hunting): picked mon.noname-a con 0x7f13b00a9c00 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 10 monclient(hunting): picked mon.noname-c con 0x7f13b00b38c0 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 10 monclient(hunting): picked mon.noname-b con 0x7f13b01df540 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b01de9a0 con 0x7f13b00a9c00 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b00b0ff0 con 0x7f13b01df540 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b0139eb0 con 0x7f13b00b38c0 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:10.345 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.362+0000 7f13affff640 10 monclient(hunting): authenticate will time out at 1253.654343s 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13af7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f13b01df540 0x7f13b01397a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:46853/0 (socket says 172.21.15.155:46853) 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13af7fe640 1 -- 172.21.15.155:0/2529863059 learned_addr learned my addr 172.21.15.155:0/2529863059 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 client.?.objecter ms_handle_connect 0x7f13b01df540 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 client.?.objecter ms_handle_connect 0x7f13b00b38c0 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705294293 0 0) 0x7f13b0139eb0 con 0x7f13b00b38c0 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:59:10.347 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.366+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139c004eb0 con 0x7f13b00b38c0 2024-03-29T10:59:10.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716846871 0 0) 0x7f139c004eb0 con 0x7f13b00b38c0 2024-03-29T10:59:10.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13b0139eb0 con 0x7f13b00b38c0 2024-03-29T10:59:10.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f13a0001870 con 0x7f13b00b38c0 2024-03-29T10:59:10.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:10.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:10.350 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 10 monclient(hunting): dump: 2024-03-29T10:59:10.350 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.367+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3219555151 0 0) 0x7f13b0139eb0 con 0x7f13b00b38c0 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.368+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 >> v1:172.21.15.183:6789/0 conn(0x7f13b01df540 legacy=0x7f13b01397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.368+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 >> v1:172.21.15.155:6789/0 conn(0x7f13b00a9c00 legacy=0x7f13b01dee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:10.351 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 1 monclient: found mon.c 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0071720 con 0x7f13b00b38c0 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 5 monclient: authenticate success, global_id 4523 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 10 monclient: _renew_subs 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b0071a00 con 0x7f13b00b38c0 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 10 monclient: _renew_subs 2024-03-29T10:59:10.352 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:10.353 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b006f680 con 0x7f13b00b38c0 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.369+0000 7f13affff640 1 librados: init done 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.372+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13a00035f0 con 0x7f13b00b38c0 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.372+0000 7f13977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.372+0000 7f13ad7fa640 4 set_mon_vals no callback set 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13affff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f13a0003900 con 0x7f13b00b38c0 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13affff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1390005080 con 0x7f13b00b38c0 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:10.354 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.373+0000 7f13977fe640 10 monclient: dump: 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:10.355 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:10.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.375+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f13a0019950 con 0x7f13b00b38c0 2024-03-29T10:59:10.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.376+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f13a00838a0 con 0x7f13b00b38c0 2024-03-29T10:59:10.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.376+0000 7f13977fe640 10 client.4523.objecter ms_dispatch 0x7f13b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:10.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.376+0000 7f13977fe640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:10.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.376+0000 7f13977fe640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:10.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.376+0000 7f13977fe640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T10:59:10.356 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.376+0000 7f13aeffd640 10 monclient: get_auth_request con 0x7f139c06dcd0 auth_method 0 2024-03-29T10:59:10.357 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.377+0000 7f13977fe640 10 client.4523.objecter ms_handle_connect 0x7f139c06dcd0 2024-03-29T10:59:10.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.382+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13a004f1e0 con 0x7f13b00b38c0 2024-03-29T10:59:10.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.382+0000 7f13977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:10.362 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.382+0000 7f13977fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:10.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.734+0000 7f13affff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:10.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.734+0000 7f13affff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:10.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.734+0000 7f13affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:10.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.734+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1390005330 con 0x7f13b00b38c0 2024-03-29T10:59:10.723 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:10.723 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.739+0000 7f13977fe640 1 -- 172.21.15.155:0/2529863059 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f13a0051110 con 0x7f13b00b38c0 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.740+0000 7f13977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.740+0000 7f13977fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.743+0000 7f13affff640 10 librados: watch_flush enter 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.743+0000 7f13affff640 10 librados: watch_flush exit 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.743+0000 7f13affff640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.743+0000 7f13affff640 10 client.4523.objecter successfully canceled tick 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.744+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 >> v1:172.21.15.183:6816/1412921548 conn(0x7f139c06dcd0 legacy=0x7f139c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:10.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.744+0000 7f13affff640 10 monclient: shutdown 2024-03-29T10:59:10.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.744+0000 7f13affff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:10.725 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.744+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 >> v1:172.21.15.155:6790/0 conn(0x7f13b00b38c0 legacy=0x7f13b0139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:10.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.751+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 shutdown_connections 2024-03-29T10:59:10.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.751+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 >> 172.21.15.155:0/2529863059 conn(0x7f13b0066a00 msgr2=0x7f13b01404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:10.731 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.751+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 shutdown_connections 2024-03-29T10:59:10.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.752+0000 7f13affff640 1 -- 172.21.15.155:0/2529863059 wait complete. 2024-03-29T10:59:10.732 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:10.753+0000 7f13affff640 1 librados: shutdown 2024-03-29T10:59:10.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:17.557 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:17.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.726+0000 7f56ab51c640 10 monclient: get_monmap_and_config 2024-03-29T10:59:17.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.726+0000 7f56ab51c640 10 monclient: build_initial_monmap 2024-03-29T10:59:17.707 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.726+0000 7f56ab51c640 10 monclient: monmap: 2024-03-29T10:59:17.707 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:17.707 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:17.707 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:17.727615+0000 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:17.727615+0000 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:17.708 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.727+0000 7f56ab51c640 1 Processor -- start 2024-03-29T10:59:17.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.728+0000 7f56ab51c640 1 -- start start 2024-03-29T10:59:17.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.728+0000 7f56ab51c640 10 monclient: init 2024-03-29T10:59:17.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.728+0000 7f56ab51c640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:17.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.728+0000 7f56ab51c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:17.709 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.728+0000 7f56ab51c640 10 monclient(hunting): picked mon.noname-c con 0x7f56a40b0860 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:17.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.729+0000 7f56ab51c640 10 monclient(hunting): picked mon.noname-a con 0x7f56a413ef30 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:17.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.730+0000 7f56ab51c640 10 monclient(hunting): picked mon.noname-b con 0x7f56a40af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:17.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.730+0000 7f56ab51c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40b0c40 con 0x7f56a413ef30 2024-03-29T10:59:17.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.730+0000 7f56ab51c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40aed10 con 0x7f56a40b0860 2024-03-29T10:59:17.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.730+0000 7f56ab51c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40afd50 con 0x7f56a40af8e0 2024-03-29T10:59:17.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.730+0000 7f56ab51c640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:17.713 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.730+0000 7f56ab51c640 10 monclient(hunting): authenticate will time out at 1261.021980s 2024-03-29T10:59:17.714 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.734+0000 7f56a8ab7640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f56a413ef30 0x7f56a40af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39145/0 (socket says 172.21.15.155:39145) 2024-03-29T10:59:17.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.734+0000 7f56a8ab7640 1 -- 172.21.15.155:0/1253604836 learned_addr learned my addr 172.21.15.155:0/1253604836 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:17.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.735+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768367824 0 0) 0x7f56a40b0c40 con 0x7f56a413ef30 2024-03-29T10:59:17.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.735+0000 7f569bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:17.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.735+0000 7f569bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:17.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.735+0000 7f569bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:17.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.735+0000 7f569bfff640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:59:17.715 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.735+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5690003520 con 0x7f56a413ef30 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690192826 0 0) 0x7f5690003520 con 0x7f56a413ef30 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 >> v1:172.21.15.155:6790/0 conn(0x7f56a40af8e0 legacy=0x7f56a4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 >> v1:172.21.15.183:6789/0 conn(0x7f56a40b0860 legacy=0x7f56a413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 1 monclient: found mon.noname-a 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a40ab1b0 con 0x7f56a413ef30 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 10 monclient: _finish_auth 0 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:17.717 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f56ab51c640 5 monclient: authenticate success, global_id 4467 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f56ab51c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f569c001a10 con 0x7f56a413ef30 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f569bfff640 10 monclient: dump: 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:17.718 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:17.719 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:17.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.737+0000 7f56ab51c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:17.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f569c001ca0 con 0x7f56a413ef30 2024-03-29T10:59:17.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f569bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:17.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f569bfff640 1 -- 172.21.15.155:0/1253604836 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f569c002030 con 0x7f56a413ef30 2024-03-29T10:59:17.719 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f569bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f569bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f569bfff640 10 monclient: dump: 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f56ab51c640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:17.720 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.739+0000 7f56ab51c640 4 set_mon_vals no callback set 2024-03-29T10:59:17.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.741+0000 7f56ab51c640 10 monclient: shutdown 2024-03-29T10:59:17.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.741+0000 7f56ab51c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:17.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.741+0000 7f56ab51c640 1 -- 172.21.15.155:0/1253604836 >> v1:172.21.15.155:6789/0 conn(0x7f56a413ef30 legacy=0x7f56a40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:17.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.741+0000 7f56ab51c640 1 -- 172.21.15.155:0/1253604836 shutdown_connections 2024-03-29T10:59:17.721 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.741+0000 7f56ab51c640 1 -- 172.21.15.155:0/1253604836 >> 172.21.15.155:0/1253604836 conn(0x7f56a406b660 msgr2=0x7f56a406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:17.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.744+0000 7f56ab51c640 1 -- 172.21.15.155:0/1253604836 shutdown_connections 2024-03-29T10:59:17.724 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.744+0000 7f56ab51c640 1 -- 172.21.15.155:0/1253604836 wait complete. 2024-03-29T10:59:17.726 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.745+0000 7f56ab51c640 10 monclient: build_initial_monmap 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.745+0000 7f56ab51c640 10 monclient: monmap: 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.745+0000 7f56ab51c640 1 librados: starting msgr at 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.745+0000 7f56ab51c640 1 librados: starting objecter 2024-03-29T10:59:17.729 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.746+0000 7f56ab51c640 1 Processor -- start 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.746+0000 7f56ab51c640 1 -- start start 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.746+0000 7f56ab51c640 1 librados: setting wanted keys 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.746+0000 7f56ab51c640 1 librados: calling monclient init 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.746+0000 7f56ab51c640 10 monclient: init 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 10 monclient(hunting): picked mon.noname-c con 0x7f56a40af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 10 monclient(hunting): picked mon.noname-a con 0x7f56a40b0860 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 10 monclient(hunting): picked mon.noname-b con 0x7f56a413ef30 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a41e7d80 con 0x7f56a40b0860 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a41c6190 con 0x7f56a413ef30 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a41c4b30 con 0x7f56a40af8e0 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:17.730 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.747+0000 7f56ab51c640 10 monclient(hunting): authenticate will time out at 1261.039091s 2024-03-29T10:59:17.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.753+0000 7f56a92b8640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f56a40af8e0 0x7f56a41e74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:52201/0 (socket says 172.21.15.155:52201) 2024-03-29T10:59:17.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.753+0000 7f56a92b8640 1 -- 172.21.15.155:0/1073357748 learned_addr learned my addr 172.21.15.155:0/1073357748 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:17.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.753+0000 7f56a9ab9640 1 --1- 172.21.15.155:0/1073357748 >> v1:172.21.15.183:6789/0 conn(0x7f56a413ef30 0x7f56a41ee780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:35539/0 (socket says 172.21.15.155:35539) 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 client.?.objecter ms_handle_connect 0x7f56a40af8e0 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 client.?.objecter ms_handle_connect 0x7f56a40b0860 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163565657 0 0) 0x7f56a41e7d80 con 0x7f56a40b0860 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5688004eb0 con 0x7f56a40b0860 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 client.?.objecter ms_handle_connect 0x7f56a413ef30 2024-03-29T10:59:17.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:17.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.754+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702371232 0 0) 0x7f56a41c4b30 con 0x7f56a40af8e0 2024-03-29T10:59:17.737 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a41e7d80 con 0x7f56a40af8e0 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2113571116 0 0) 0x7f5688004eb0 con 0x7f56a40b0860 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56a41c4b30 con 0x7f56a40b0860 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f569c0018d0 con 0x7f56a40b0860 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:17.738 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790527015 0 0) 0x7f56a41e7d80 con 0x7f56a40af8e0 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.755+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5688004eb0 con 0x7f56a40af8e0 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f56a0003f00 con 0x7f56a40af8e0 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:17.739 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026941864 0 0) 0x7f56a41c6190 con 0x7f56a413ef30 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:17.740 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a41e7d80 con 0x7f56a413ef30 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 878775808 0 0) 0x7f5688004eb0 con 0x7f56a40af8e0 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 >> v1:172.21.15.183:6789/0 conn(0x7f56a413ef30 legacy=0x7f56a41ee780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 >> v1:172.21.15.155:6789/0 conn(0x7f56a40b0860 legacy=0x7f56a41c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 monclient: found mon.c 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a41c4cc0 con 0x7f56a40af8e0 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:17.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:17.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.756+0000 7f5699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:17.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:59:17.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 10 monclient: _renew_subs 2024-03-29T10:59:17.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:17.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a41c4f70 con 0x7f56a40af8e0 2024-03-29T10:59:17.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:17.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 10 monclient: _renew_subs 2024-03-29T10:59:17.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:17.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a40b4120 con 0x7f56a40af8e0 2024-03-29T10:59:17.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.757+0000 7f56ab51c640 1 librados: init done 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.759+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56a00044d0 con 0x7f56a40af8e0 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.759+0000 7f5699ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.759+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f56a00040a0 con 0x7f56a40af8e0 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.759+0000 7f5699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.759+0000 7f5699ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.759+0000 7f5699ffb640 10 monclient: dump: 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:17.745 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.759+0000 7f569b7fe640 4 set_mon_vals no callback set 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.761+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f56a0001170 con 0x7f56a40af8e0 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f56ab51c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f56ab51c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f56ab51c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5684005080 con 0x7f56a40af8e0 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f56a8ab7640 10 monclient: get_auth_request con 0x7f568806e0d0 auth_method 0 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f56a0084880 con 0x7f56a40af8e0 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f5699ffb640 10 client.4529.objecter ms_dispatch 0x7f56a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f5699ffb640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f5699ffb640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.762+0000 7f5699ffb640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T10:59:17.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.763+0000 7f5699ffb640 10 client.4529.objecter ms_handle_connect 0x7f568806e0d0 2024-03-29T10:59:17.750 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.770+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56a0050160 con 0x7f56a40af8e0 2024-03-29T10:59:17.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.770+0000 7f5699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:17.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:17.770+0000 7f5699ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:18.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.096+0000 7f56ab51c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.096+0000 7f56ab51c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.096+0000 7f56ab51c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:18.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.096+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5684005330 con 0x7f56a40af8e0 2024-03-29T10:59:18.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.097+0000 7f5699ffb640 1 -- 172.21.15.155:0/1073357748 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f56a0052090 con 0x7f56a40af8e0 2024-03-29T10:59:18.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.097+0000 7f5699ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:18.077 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.097+0000 7f5699ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:18.078 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:18.078 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:18.080 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 10 librados: watch_flush enter 2024-03-29T10:59:18.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 10 librados: watch_flush exit 2024-03-29T10:59:18.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T10:59:18.081 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 10 client.4529.objecter successfully canceled tick 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 >> v1:172.21.15.183:6816/1412921548 conn(0x7f568806e0d0 legacy=0x7f5688070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 10 monclient: shutdown 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 >> v1:172.21.15.155:6790/0 conn(0x7f56a40af8e0 legacy=0x7f56a41e74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 shutdown_connections 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.101+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 >> 172.21.15.155:0/1073357748 conn(0x7f56a406b660 msgr2=0x7f56a413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.102+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 shutdown_connections 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.102+0000 7f56ab51c640 1 -- 172.21.15.155:0/1073357748 wait complete. 2024-03-29T10:59:18.082 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:18.102+0000 7f56ab51c640 1 librados: shutdown 2024-03-29T10:59:18.100 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:24.907 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:25.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.074+0000 7f0b6516b640 10 monclient: get_monmap_and_config 2024-03-29T10:59:25.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.074+0000 7f0b6516b640 10 monclient: build_initial_monmap 2024-03-29T10:59:25.054 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.074+0000 7f0b6516b640 10 monclient: monmap: 2024-03-29T10:59:25.054 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:25.075065+0000 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:25.075065+0000 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:25.055 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.075+0000 7f0b6516b640 1 Processor -- start 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.075+0000 7f0b6516b640 1 -- start start 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.075+0000 7f0b6516b640 10 monclient: init 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b60071c30 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b600a96c0 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:25.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6013e1e0 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:25.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600ab300 con 0x7f0b6013e1e0 2024-03-29T10:59:25.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600a9b00 con 0x7f0b600a96c0 2024-03-29T10:59:25.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600ac2f0 con 0x7f0b60071c30 2024-03-29T10:59:25.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:25.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b6516b640 10 monclient(hunting): authenticate will time out at 1268.368295s 2024-03-29T10:59:25.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b5ed9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f0b60071c30 0x7f0b600b0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40510/0 (socket says 172.21.15.155:40510) 2024-03-29T10:59:25.057 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.076+0000 7f0b5ed9d640 1 -- 172.21.15.155:0/3034750992 learned_addr learned my addr 172.21.15.155:0/3034750992 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:25.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334023683 0 0) 0x7f0b600ac2f0 con 0x7f0b60071c30 2024-03-29T10:59:25.059 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c003520 con 0x7f0b60071c30 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540565561 0 0) 0x7f0b600ab300 con 0x7f0b6013e1e0 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b600ac2f0 con 0x7f0b6013e1e0 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243252601 0 0) 0x7f0b4c003520 con 0x7f0b60071c30 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 >> v1:172.21.15.183:6789/0 conn(0x7f0b600a96c0 legacy=0x7f0b6013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 >> v1:172.21.15.155:6789/0 conn(0x7f0b6013e1e0 legacy=0x7f0b600ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.060 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 monclient: found mon.noname-b 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b60145ec0 con 0x7f0b60071c30 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0b50001690 con 0x7f0b60071c30 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: dump: 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:25.061 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b6516b640 5 monclient: authenticate success, global_id 4532 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b6516b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b500017f0 con 0x7f0b60071c30 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 1 -- 172.21.15.155:0/3034750992 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0b50001c00 con 0x7f0b60071c30 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.062 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.077+0000 7f0b5dd9b640 10 monclient: dump: 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.078+0000 7f0b6516b640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.078+0000 7f0b6516b640 4 set_mon_vals no callback set 2024-03-29T10:59:25.063 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.078+0000 7f0b6516b640 10 monclient: shutdown 2024-03-29T10:59:25.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.078+0000 7f0b6516b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:25.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.078+0000 7f0b6516b640 1 -- 172.21.15.155:0/3034750992 >> v1:172.21.15.155:6790/0 conn(0x7f0b60071c30 legacy=0x7f0b600b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.079+0000 7f0b6516b640 1 -- 172.21.15.155:0/3034750992 shutdown_connections 2024-03-29T10:59:25.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.079+0000 7f0b6516b640 1 -- 172.21.15.155:0/3034750992 >> 172.21.15.155:0/3034750992 conn(0x7f0b6006b660 msgr2=0x7f0b6006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:25.064 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.079+0000 7f0b6516b640 1 -- 172.21.15.155:0/3034750992 shutdown_connections 2024-03-29T10:59:25.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.079+0000 7f0b6516b640 1 -- 172.21.15.155:0/3034750992 wait complete. 2024-03-29T10:59:25.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.081+0000 7f0b6516b640 10 monclient: build_initial_monmap 2024-03-29T10:59:25.067 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.081+0000 7f0b6516b640 10 monclient: monmap: 2024-03-29T10:59:25.067 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.081+0000 7f0b6516b640 1 librados: starting msgr at 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.081+0000 7f0b6516b640 1 librados: starting objecter 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.081+0000 7f0b6516b640 1 Processor -- start 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 1 -- start start 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 1 librados: setting wanted keys 2024-03-29T10:59:25.068 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 1 librados: calling monclient init 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient: init 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b60071c30 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b600a96c0 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b6013e1e0 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6006f0f0 con 0x7f0b600a96c0 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b601db7f0 con 0x7f0b60071c30 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6006f360 con 0x7f0b6013e1e0 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.082+0000 7f0b6516b640 10 monclient(hunting): authenticate will time out at 1268.374798s 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b5ed9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f0b60071c30 0x7f0b6006dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:58916/0 (socket says 172.21.15.155:58916) 2024-03-29T10:59:25.069 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b5ed9d640 1 -- 172.21.15.155:0/1962816563 learned_addr learned my addr 172.21.15.155:0/1962816563 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 client.?.objecter ms_handle_connect 0x7f0b60071c30 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 client.?.objecter ms_handle_connect 0x7f0b6013e1e0 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851125537 0 0) 0x7f0b601db7f0 con 0x7f0b60071c30 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.083+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40004eb0 con 0x7f0b60071c30 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 client.?.objecter ms_handle_connect 0x7f0b600a96c0 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2935076344 0 0) 0x7f0b40004eb0 con 0x7f0b60071c30 2024-03-29T10:59:25.070 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b601db7f0 con 0x7f0b60071c30 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541236024 0 0) 0x7f0b6006f360 con 0x7f0b6013e1e0 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40004eb0 con 0x7f0b6013e1e0 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0b50001c00 con 0x7f0b60071c30 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): dump: 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.071 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 852544570 0 0) 0x7f0b601db7f0 con 0x7f0b60071c30 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 >> v1:172.21.15.155:6790/0 conn(0x7f0b6013e1e0 legacy=0x7f0b601e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 >> v1:172.21.15.155:6789/0 conn(0x7f0b600a96c0 legacy=0x7f0b601dbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 monclient: found mon.b 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:25.072 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6012dbc0 con 0x7f0b60071c30 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 5 monclient: authenticate success, global_id 4465 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 10 monclient: _renew_subs 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b601dc3e0 con 0x7f0b60071c30 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 10 monclient: _renew_subs 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b601308c0 con 0x7f0b60071c30 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.084+0000 7f0b6516b640 1 librados: init done 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.085+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b50004a90 con 0x7f0b60071c30 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.085+0000 7f0b577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:25.073 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.085+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f0b50004e60 con 0x7f0b60071c30 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.085+0000 7f0b577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.085+0000 7f0b577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.085+0000 7f0b577fe640 10 monclient: dump: 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:25.074 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.085+0000 7f0b5d59a640 4 set_mon_vals no callback set 2024-03-29T10:59:25.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.087+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f0b50004fc0 con 0x7f0b60071c30 2024-03-29T10:59:25.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.087+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f0b50084bc0 con 0x7f0b60071c30 2024-03-29T10:59:25.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.087+0000 7f0b577fe640 10 client.4465.objecter ms_dispatch 0x7f0b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:25.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.087+0000 7f0b577fe640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:25.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.087+0000 7f0b577fe640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:25.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.087+0000 7f0b577fe640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T10:59:25.075 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.088+0000 7f0b6516b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.088+0000 7f0b6516b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.088+0000 7f0b6516b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.088+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b44005080 con 0x7f0b60071c30 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.088+0000 7f0b5e59c640 10 monclient: get_auth_request con 0x7f0b4006de30 auth_method 0 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.089+0000 7f0b577fe640 10 client.4465.objecter ms_handle_connect 0x7f0b4006de30 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.096+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b500504a0 con 0x7f0b60071c30 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.096+0000 7f0b577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:25.076 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.096+0000 7f0b577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:25.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.416+0000 7f0b6516b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:25.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.416+0000 7f0b6516b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:25.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.416+0000 7f0b6516b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:25.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.416+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b44005330 con 0x7f0b60071c30 2024-03-29T10:59:25.398 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:25.399 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.417+0000 7f0b577fe640 1 -- 172.21.15.155:0/1962816563 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f0b500523d0 con 0x7f0b60071c30 2024-03-29T10:59:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.417+0000 7f0b577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:25.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.417+0000 7f0b577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 10 librados: watch_flush enter 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 10 librados: watch_flush exit 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 10 client.4465.objecter successfully canceled tick 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 >> v1:172.21.15.183:6816/1412921548 conn(0x7f0b4006de30 legacy=0x7f0b400702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 10 monclient: shutdown 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:25.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.421+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 >> v1:172.21.15.183:6789/0 conn(0x7f0b60071c30 legacy=0x7f0b6006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:25.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.422+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 shutdown_connections 2024-03-29T10:59:25.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.422+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 >> 172.21.15.155:0/1962816563 conn(0x7f0b6006b660 msgr2=0x7f0b6006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:25.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.423+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 shutdown_connections 2024-03-29T10:59:25.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.423+0000 7f0b6516b640 1 -- 172.21.15.155:0/1962816563 wait complete. 2024-03-29T10:59:25.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:25.423+0000 7f0b6516b640 1 librados: shutdown 2024-03-29T10:59:25.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:32.224 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:32.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.406+0000 7f20d0af6640 10 monclient: get_monmap_and_config 2024-03-29T10:59:32.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.406+0000 7f20d0af6640 10 monclient: build_initial_monmap 2024-03-29T10:59:32.387 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.406+0000 7f20d0af6640 10 monclient: monmap: 2024-03-29T10:59:32.387 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:32.387 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:32.407362+0000 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:32.407362+0000 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:32.388 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.407+0000 7f20d0af6640 1 Processor -- start 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.407+0000 7f20d0af6640 1 -- start start 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.407+0000 7f20d0af6640 10 monclient: init 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.408+0000 7f20d0af6640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.408+0000 7f20d0af6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 10 monclient(hunting): picked mon.noname-c con 0x7f20cc0a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 10 monclient(hunting): picked mon.noname-a con 0x7f20cc13ee90 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 10 monclient(hunting): picked mon.noname-b con 0x7f20cc0af900 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc0afd10 con 0x7f20cc13ee90 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc0af6f0 con 0x7f20cc0a9ac0 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc0aac80 con 0x7f20cc0af900 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20d0af6640 10 monclient(hunting): authenticate will time out at 1275.700998s 2024-03-29T10:59:32.389 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20cad9d640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f20cc0af900 0x7f20cc149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:49086/0 (socket says 172.21.15.155:49086) 2024-03-29T10:59:32.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20cad9d640 1 -- 172.21.15.155:0/1215419677 learned_addr learned my addr 172.21.15.155:0/1215419677 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:32.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816439813 0 0) 0x7f20cc0aac80 con 0x7f20cc0af900 2024-03-29T10:59:32.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20c959a640 10 monclient(hunting): _negotiate 2024-03-29T10:59:32.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:32.390 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20c959a640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:59:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.409+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b4003520 con 0x7f20cc0af900 2024-03-29T10:59:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.410+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3936383413 0 0) 0x7f20b4003520 con 0x7f20cc0af900 2024-03-29T10:59:32.391 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.410+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 >> v1:172.21.15.183:6789/0 conn(0x7f20cc0a9ac0 legacy=0x7f20cc0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:32.392 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.413+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 >> v1:172.21.15.155:6789/0 conn(0x7f20cc13ee90 legacy=0x7f20cc13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 10 monclient: _finish_hunting 0 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 1 monclient: found mon.noname-b 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc0aaf60 con 0x7f20cc0af900 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 10 monclient: _finish_auth 0 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 10 monclient: _check_auth_tickets 2024-03-29T10:59:32.396 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20bc001690 con 0x7f20cc0af900 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20c959a640 10 monclient: dump: 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:32.397 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:32.398 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:32.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20d0af6640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:59:32.398 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.416+0000 7f20d0af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20bc001860 con 0x7f20cc0af900 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20c959a640 1 -- 172.21.15.155:0/1215419677 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20bc001cb0 con 0x7f20cc0af900 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20c959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20c959a640 10 monclient: dump: 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:32.399 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20d0af6640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.417+0000 7f20d0af6640 4 set_mon_vals no callback set 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.418+0000 7f20d0af6640 10 monclient: shutdown 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.418+0000 7f20d0af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.418+0000 7f20d0af6640 1 -- 172.21.15.155:0/1215419677 >> v1:172.21.15.155:6790/0 conn(0x7f20cc0af900 legacy=0x7f20cc149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.418+0000 7f20d0af6640 1 -- 172.21.15.155:0/1215419677 shutdown_connections 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.418+0000 7f20d0af6640 1 -- 172.21.15.155:0/1215419677 >> 172.21.15.155:0/1215419677 conn(0x7f20cc0b35e0 msgr2=0x7f20cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:32.400 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.418+0000 7f20d0af6640 1 -- 172.21.15.155:0/1215419677 shutdown_connections 2024-03-29T10:59:32.401 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.418+0000 7f20d0af6640 1 -- 172.21.15.155:0/1215419677 wait complete. 2024-03-29T10:59:32.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.419+0000 7f20d0af6640 10 monclient: build_initial_monmap 2024-03-29T10:59:32.402 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.419+0000 7f20d0af6640 10 monclient: monmap: 2024-03-29T10:59:32.402 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:32.402 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:32.402 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:32.402 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:32.402 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.420+0000 7f20d0af6640 1 librados: starting msgr at 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.420+0000 7f20d0af6640 1 librados: starting objecter 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.420+0000 7f20d0af6640 1 Processor -- start 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.420+0000 7f20d0af6640 1 -- start start 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.420+0000 7f20d0af6640 1 librados: setting wanted keys 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.420+0000 7f20d0af6640 1 librados: calling monclient init 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.420+0000 7f20d0af6640 10 monclient: init 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.421+0000 7f20d0af6640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.421+0000 7f20d0af6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:32.403 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.421+0000 7f20d0af6640 10 monclient(hunting): picked mon.noname-c con 0x7f20cc0a9ac0 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20d0af6640 10 monclient(hunting): picked mon.noname-b con 0x7f20cc0af900 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20d0af6640 10 monclient(hunting): picked mon.noname-a con 0x7f20cc13ee90 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20d0af6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc072380 con 0x7f20cc13ee90 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20d0af6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc071a30 con 0x7f20cc0af900 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20d0af6640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc071c80 con 0x7f20cc0a9ac0 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20d0af6640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20d0af6640 10 monclient(hunting): authenticate will time out at 1275.716411s 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.424+0000 7f20c9d9b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f20cc0af900 0x7f20cc1410b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:54631/0 (socket says 172.21.15.155:54631) 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20c9d9b640 1 -- 172.21.15.155:0/3954205275 learned_addr learned my addr 172.21.15.155:0/3954205275 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:32.407 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 client.?.objecter ms_handle_connect 0x7f20cc0a9ac0 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 client.?.objecter ms_handle_connect 0x7f20cc0af900 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874111398 0 0) 0x7f20cc071a30 con 0x7f20cc0af900 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.425+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ac004eb0 con 0x7f20cc0af900 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563729673 0 0) 0x7f20cc071c80 con 0x7f20cc0a9ac0 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20cc071a30 con 0x7f20cc0a9ac0 2024-03-29T10:59:32.408 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516562693 0 0) 0x7f20ac004eb0 con 0x7f20cc0af900 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20cc071c80 con 0x7f20cc0af900 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20c0003f10 con 0x7f20cc0af900 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): dump: 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:32.409 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.426+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2426897723 0 0) 0x7f20cc071c80 con 0x7f20cc0af900 2024-03-29T10:59:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.427+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 >> v1:172.21.15.155:6790/0 conn(0x7f20cc0a9ac0 legacy=0x7f20cc0706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:32.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.427+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 >> v1:172.21.15.155:6789/0 conn(0x7f20cc13ee90 legacy=0x7f20cc1417c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:32.411 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.428+0000 7f20baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.428+0000 7f20baffd640 1 monclient: found mon.b 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.428+0000 7f20baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20baffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc071f60 con 0x7f20cc0af900 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20baffd640 10 monclient: _finish_auth 0 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 5 monclient: authenticate success, global_id 4468 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 10 monclient: _renew_subs 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc1386f0 con 0x7f20cc0af900 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:32.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 10 monclient: _renew_subs 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc138bd0 con 0x7f20cc0af900 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.429+0000 7f20d0af6640 1 librados: init done 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.430+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20c0004070 con 0x7f20cc0af900 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.430+0000 7f20baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.430+0000 7f20c8d99640 4 set_mon_vals no callback set 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.430+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20c0001070 con 0x7f20cc0af900 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.430+0000 7f20baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.430+0000 7f20baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.430+0000 7f20baffd640 10 monclient: dump: 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:32.413 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:32.414 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:32.414 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:32.414 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:32.414 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:32.414 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.432+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f20c00011d0 con 0x7f20cc0af900 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f20c0084930 con 0x7f20cc0af900 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20baffd640 10 client.4468.objecter ms_dispatch 0x7f20cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20baffd640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20baffd640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20baffd640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20ca59c640 10 monclient: get_auth_request con 0x7f20ac06ddc0 auth_method 0 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20d0af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20d0af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20d0af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.433+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b0005080 con 0x7f20cc0af900 2024-03-29T10:59:32.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.434+0000 7f20baffd640 10 client.4468.objecter ms_handle_connect 0x7f20ac06ddc0 2024-03-29T10:59:32.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.439+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20c0050210 con 0x7f20cc0af900 2024-03-29T10:59:32.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.440+0000 7f20baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:32.419 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.440+0000 7f20baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:32.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.809+0000 7f20d0af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:32.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.809+0000 7f20d0af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:32.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.809+0000 7f20d0af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:32.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.809+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20b0005330 con 0x7f20cc0af900 2024-03-29T10:59:32.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.810+0000 7f20baffd640 1 -- 172.21.15.155:0/3954205275 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f20c0052140 con 0x7f20cc0af900 2024-03-29T10:59:32.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.810+0000 7f20baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:32.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.810+0000 7f20baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:32.791 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:32.791 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 10 librados: watch_flush enter 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 10 librados: watch_flush exit 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 10 client.4468.objecter successfully canceled tick 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 >> v1:172.21.15.183:6816/1412921548 conn(0x7f20ac06ddc0 legacy=0x7f20ac070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 10 monclient: shutdown 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:32.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 >> v1:172.21.15.183:6789/0 conn(0x7f20cc0af900 legacy=0x7f20cc1410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:32.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 shutdown_connections 2024-03-29T10:59:32.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.814+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 >> 172.21.15.155:0/3954205275 conn(0x7f20cc0b35e0 msgr2=0x7f20cc1ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:32.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.815+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 shutdown_connections 2024-03-29T10:59:32.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.815+0000 7f20d0af6640 1 -- 172.21.15.155:0/3954205275 wait complete. 2024-03-29T10:59:32.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:32.815+0000 7f20d0af6640 1 librados: shutdown 2024-03-29T10:59:32.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:39.617 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:39.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.783+0000 7f5fa1a74640 10 monclient: get_monmap_and_config 2024-03-29T10:59:39.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.783+0000 7f5fa1a74640 10 monclient: build_initial_monmap 2024-03-29T10:59:39.764 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.783+0000 7f5fa1a74640 10 monclient: monmap: 2024-03-29T10:59:39.764 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:39.764 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:39.784924+0000 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:39.784924+0000 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.784+0000 7f5fa1a74640 1 Processor -- start 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.784+0000 7f5fa1a74640 1 -- start start 2024-03-29T10:59:39.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.784+0000 7f5fa1a74640 10 monclient: init 2024-03-29T10:59:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9c0b1f60 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:39.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9c071eb0 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9c143d10 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c06ef80 con 0x7f5f9c071eb0 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c06f260 con 0x7f5f9c143d10 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0b40c0 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5fa1a74640 10 monclient(hunting): authenticate will time out at 1283.078434s 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5f9b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f5f9c0b1f60 0x7f5f9c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:54968/0 (socket says 172.21.15.155:54968) 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5f9b7fe640 1 -- 172.21.15.155:0/2342399959 learned_addr learned my addr 172.21.15.155:0/2342399959 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:39.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.785+0000 7f5f9affd640 1 --1- 172.21.15.155:0/2342399959 >> v1:172.21.15.155:6789/0 conn(0x7f5f9c071eb0 0x7f5f9c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43358/0 (socket says 172.21.15.155:43358) 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733476193 0 0) 0x7f5f9c0b40c0 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f84003520 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 642889742 0 0) 0x7f5f84003520 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 >> v1:172.21.15.183:6789/0 conn(0x7f5f9c143d10 legacy=0x7f5f9c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 >> v1:172.21.15.155:6789/0 conn(0x7f5f9c071eb0 legacy=0x7f5f9c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c155d70 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5fa1a74640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5fa1a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5f8c001670 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:39.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: dump: 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5fa1a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f8c001850 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.786+0000 7f5f9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:39.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.787+0000 7f5f9a7fc640 1 -- 172.21.15.155:0/2342399959 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5f8c001670 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.787+0000 7f5f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.787+0000 7f5f9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.787+0000 7f5f9a7fc640 10 monclient: dump: 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.787+0000 7f5fa1a74640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:39.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.787+0000 7f5fa1a74640 4 set_mon_vals no callback set 2024-03-29T10:59:39.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.788+0000 7f5fa1a74640 10 monclient: shutdown 2024-03-29T10:59:39.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.788+0000 7f5fa1a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:39.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.788+0000 7f5fa1a74640 1 -- 172.21.15.155:0/2342399959 >> v1:172.21.15.155:6790/0 conn(0x7f5f9c0b1f60 legacy=0x7f5f9c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:39.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.788+0000 7f5fa1a74640 1 -- 172.21.15.155:0/2342399959 shutdown_connections 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.788+0000 7f5fa1a74640 1 -- 172.21.15.155:0/2342399959 >> 172.21.15.155:0/2342399959 conn(0x7f5f9c066a00 msgr2=0x7f5f9c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.788+0000 7f5fa1a74640 1 -- 172.21.15.155:0/2342399959 shutdown_connections 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.788+0000 7f5fa1a74640 1 -- 172.21.15.155:0/2342399959 wait complete. 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.789+0000 7f5fa1a74640 10 monclient: build_initial_monmap 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.789+0000 7f5fa1a74640 10 monclient: monmap: 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:39.775 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.789+0000 7f5fa1a74640 1 librados: starting msgr at 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.789+0000 7f5fa1a74640 1 librados: starting objecter 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.789+0000 7f5fa1a74640 1 Processor -- start 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.790+0000 7f5fa1a74640 1 -- start start 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.790+0000 7f5fa1a74640 1 librados: setting wanted keys 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.790+0000 7f5fa1a74640 1 librados: calling monclient init 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.790+0000 7f5fa1a74640 10 monclient: init 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9c0b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f9b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f5f9c0b1f60 0x7f5f9c147ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:49136/0 (socket says 172.21.15.155:49136) 2024-03-29T10:59:39.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f9b7fe640 1 -- 172.21.15.155:0/898011261 learned_addr learned my addr 172.21.15.155:0/898011261 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9c143d10 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9c154a00 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c14b020 con 0x7f5f9c154a00 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c14b300 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c148450 con 0x7f5f9c143d10 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5fa1a74640 10 monclient(hunting): authenticate will time out at 1283.084349s 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 client.?.objecter ms_handle_connect 0x7f5f9c143d10 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 client.?.objecter ms_handle_connect 0x7f5f9c154a00 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 client.?.objecter ms_handle_connect 0x7f5f9c0b1f60 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:39.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035977569 0 0) 0x7f5f9c14b300 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.791+0000 7f5f83fff640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f88004eb0 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853697289 0 0) 0x7f5f9c148450 con 0x7f5f9c143d10 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9c14b300 con 0x7f5f9c143d10 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754824587 0 0) 0x7f5f9c14b020 con 0x7f5f9c154a00 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:39.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9c148450 con 0x7f5f9c154a00 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525605566 0 0) 0x7f5f88004eb0 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f9c14b020 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5f8c001670 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): dump: 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.779 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 544031543 0 0) 0x7f5f9c14b300 con 0x7f5f9c143d10 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f88004eb0 con 0x7f5f9c143d10 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5f90001820 con 0x7f5f9c143d10 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 10 monclient(hunting): dump: 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:39.780 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304777179 0 0) 0x7f5f9c148450 con 0x7f5f9c154a00 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.792+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f9c14b300 con 0x7f5f9c154a00 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5f94001ad0 con 0x7f5f9c154a00 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:39.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 10 monclient(hunting): dump: 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3065181099 0 0) 0x7f5f9c14b020 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 >> v1:172.21.15.155:6790/0 conn(0x7f5f9c143d10 legacy=0x7f5f9c1deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 >> v1:172.21.15.155:6789/0 conn(0x7f5f9c154a00 legacy=0x7f5f9c1df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:39.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 1 monclient: found mon.b 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c1ea130 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5f83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 10 monclient: _renew_subs 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c1e7100 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:39.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 10 monclient: _renew_subs 2024-03-29T10:59:39.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:39.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c1e7590 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.793+0000 7f5fa1a74640 1 librados: init done 2024-03-29T10:59:39.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.794+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f8c0011d0 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.794+0000 7f5f83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:39.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.794+0000 7f5f99ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5fa1a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5f8c004df0 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5f83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5f83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5f83fff640 10 monclient: dump: 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:39.786 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5fa1a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5fa1a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.795+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c075f10 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.797+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f5f8c01adc0 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.797+0000 7f5f9affd640 10 monclient: get_auth_request con 0x7f5f8806e0f0 auth_method 0 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.797+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f5f8c084c60 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.797+0000 7f5f83fff640 10 client.4474.objecter ms_dispatch 0x7f5f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.798+0000 7f5f83fff640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.798+0000 7f5f83fff640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.798+0000 7f5f83fff640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:59:39.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.798+0000 7f5f83fff640 10 client.4474.objecter ms_handle_connect 0x7f5f8806e0f0 2024-03-29T10:59:39.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.806+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f8c050540 con 0x7f5f9c0b1f60 2024-03-29T10:59:39.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.806+0000 7f5f83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:39.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:39.806+0000 7f5f83fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:40.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.179+0000 7f5fa1a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:40.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.179+0000 7f5fa1a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:40.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.179+0000 7f5fa1a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T10:59:40.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.179+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f9c0b86d0 con 0x7f5f9c0b1f60 2024-03-29T10:59:40.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.180+0000 7f5f83fff640 1 -- 172.21.15.155:0/898011261 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f5f8c052470 con 0x7f5f9c0b1f60 2024-03-29T10:59:40.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.180+0000 7f5f83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:40.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.180+0000 7f5f83fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:40.162 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:40.162 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:40.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.184+0000 7f5fa1a74640 10 librados: watch_flush enter 2024-03-29T10:59:40.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.184+0000 7f5fa1a74640 10 librados: watch_flush exit 2024-03-29T10:59:40.165 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.184+0000 7f5fa1a74640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:59:40.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.184+0000 7f5fa1a74640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:59:40.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 >> v1:172.21.15.183:6816/1412921548 conn(0x7f5f8806e0f0 legacy=0x7f5f880705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 10 monclient: shutdown 2024-03-29T10:59:40.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:40.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 >> v1:172.21.15.183:6789/0 conn(0x7f5f9c0b1f60 legacy=0x7f5f9c147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:40.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 shutdown_connections 2024-03-29T10:59:40.166 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 >> 172.21.15.155:0/898011261 conn(0x7f5f9c066a00 msgr2=0x7f5f9c142b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:40.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 shutdown_connections 2024-03-29T10:59:40.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 1 -- 172.21.15.155:0/898011261 wait complete. 2024-03-29T10:59:40.167 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:40.185+0000 7f5fa1a74640 1 librados: shutdown 2024-03-29T10:59:40.180 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:46.987 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:47.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.140+0000 7f6ead98f640 10 monclient: get_monmap_and_config 2024-03-29T10:59:47.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.140+0000 7f6ead98f640 10 monclient: build_initial_monmap 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.140+0000 7f6ead98f640 10 monclient: monmap: 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:47.142389+0000 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:47.142389+0000 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:47.122 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.141+0000 7f6ead98f640 1 Processor -- start 2024-03-29T10:59:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 1 -- start start 2024-03-29T10:59:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient: init 2024-03-29T10:59:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:47.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea81457f0 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea813f650 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea80a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80aaef0 con 0x7f6ea813f650 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea8145bd0 con 0x7f6ea80a9d70 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea8145eb0 con 0x7f6ea81457f0 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.142+0000 7f6ead98f640 10 monclient(hunting): authenticate will time out at 1290.435865s 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea77fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f6ea80a9d70 0x7f6ea814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:58862/0 (socket says 172.21.15.155:58862) 2024-03-29T10:59:47.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea77fe640 1 -- 172.21.15.155:0/51186860 learned_addr learned my addr 172.21.15.155:0/51186860 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924877829 0 0) 0x7f6ea8145eb0 con 0x7f6ea81457f0 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e94003520 con 0x7f6ea81457f0 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401155964 0 0) 0x7f6e94003520 con 0x7f6ea81457f0 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 >> v1:172.21.15.183:6789/0 conn(0x7f6ea80a9d70 legacy=0x7f6ea814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:47.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.143+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 >> v1:172.21.15.155:6789/0 conn(0x7f6ea813f650 legacy=0x7f6ea80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 1 monclient: found mon.noname-b 2024-03-29T10:59:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:59:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea8146340 con 0x7f6ea81457f0 2024-03-29T10:59:47.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6e98001690 con 0x7f6ea81457f0 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: dump: 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:47.127 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:47.128 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ead98f640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:59:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ead98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e98001860 con 0x7f6ea81457f0 2024-03-29T10:59:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ea67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:47.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ead98f640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.145+0000 7f6ead98f640 4 set_mon_vals no callback set 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.146+0000 7f6ea67fc640 1 -- 172.21.15.155:0/51186860 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6e98001cb0 con 0x7f6ea81457f0 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.146+0000 7f6ea67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.146+0000 7f6ea67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.146+0000 7f6ea67fc640 10 monclient: dump: 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:47.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.146+0000 7f6ead98f640 10 monclient: shutdown 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.146+0000 7f6ead98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.146+0000 7f6ead98f640 1 -- 172.21.15.155:0/51186860 >> v1:172.21.15.155:6790/0 conn(0x7f6ea81457f0 legacy=0x7f6ea813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.147+0000 7f6ead98f640 1 -- 172.21.15.155:0/51186860 shutdown_connections 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.147+0000 7f6ead98f640 1 -- 172.21.15.155:0/51186860 >> 172.21.15.155:0/51186860 conn(0x7f6ea8066a00 msgr2=0x7f6ea8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.147+0000 7f6ead98f640 1 -- 172.21.15.155:0/51186860 shutdown_connections 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.147+0000 7f6ead98f640 1 -- 172.21.15.155:0/51186860 wait complete. 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 10 monclient: build_initial_monmap 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 10 monclient: monmap: 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:47.130 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:47.131 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:47.131 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:47.131 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:47.131 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 1 librados: starting msgr at 2024-03-29T10:59:47.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 1 librados: starting objecter 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 1 Processor -- start 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 1 -- start start 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 1 librados: setting wanted keys 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 1 librados: calling monclient init 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.148+0000 7f6ead98f640 10 monclient: init 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea80a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea813f650 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea81457f0 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80acf80 con 0x7f6ea81457f0 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea81d6f80 con 0x7f6ea80a9d70 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea81d7e00 con 0x7f6ea813f650 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:47.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.150+0000 7f6ead98f640 10 monclient(hunting): authenticate will time out at 1290.443475s 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6e9ffff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f6ea813f650 0x7f6ea8137d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39384/0 (socket says 172.21.15.155:39384) 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6e9ffff640 1 -- 172.21.15.155:0/186501947 learned_addr learned my addr 172.21.15.155:0/186501947 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 client.?.objecter ms_handle_connect 0x7f6ea813f650 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 client.?.objecter ms_handle_connect 0x7f6ea80a9d70 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856173384 0 0) 0x7f6ea81d7e00 con 0x7f6ea813f650 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e88004eb0 con 0x7f6ea813f650 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3011972766 0 0) 0x7f6e88004eb0 con 0x7f6ea813f650 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ea81d7e00 con 0x7f6ea813f650 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6e90001960 con 0x7f6ea813f650 2024-03-29T10:59:47.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.151+0000 7f6ea4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190881317 0 0) 0x7f6ea81d6f80 con 0x7f6ea80a9d70 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:47.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e88004eb0 con 0x7f6ea80a9d70 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4041187232 0 0) 0x7f6ea81d7e00 con 0x7f6ea813f650 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 >> v1:172.21.15.183:6789/0 conn(0x7f6ea80a9d70 legacy=0x7f6ea81d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.152+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 >> v1:172.21.15.155:6789/0 conn(0x7f6ea81457f0 legacy=0x7f6ea81e2800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 1 monclient: found mon.c 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea80ad1d0 con 0x7f6ea813f650 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:59:47.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:59:47.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ea4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:47.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:59:47.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 10 monclient: _renew_subs 2024-03-29T10:59:47.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:47.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea81d71d0 con 0x7f6ea813f650 2024-03-29T10:59:47.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:47.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 10 monclient: _renew_subs 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea8138650 con 0x7f6ea813f650 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.153+0000 7f6ead98f640 1 librados: init done 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.154+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e90001cc0 con 0x7f6ea813f650 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.154+0000 7f6ea4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.154+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6e90003a40 con 0x7f6ea813f650 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.154+0000 7f6ea4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.154+0000 7f6ea4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.154+0000 7f6ea4ff9640 10 monclient: dump: 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:47.140 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.154+0000 7f6ea57fa640 4 set_mon_vals no callback set 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.155+0000 7f6ead98f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.155+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f6e90003ba0 con 0x7f6ea813f650 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ead98f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ead98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c005080 con 0x7f6ea813f650 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f6e90084ba0 con 0x7f6ea813f650 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ea4ff9640 10 client.4553.objecter ms_dispatch 0x7f6ea8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ea4ff9640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ea4ff9640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ea4ff9640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.156+0000 7f6ea6ffd640 10 monclient: get_auth_request con 0x7f6e8806dd40 auth_method 0 2024-03-29T10:59:47.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.157+0000 7f6ea4ff9640 10 client.4553.objecter ms_handle_connect 0x7f6e8806dd40 2024-03-29T10:59:47.144 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.164+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e90050480 con 0x7f6ea813f650 2024-03-29T10:59:47.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.164+0000 7f6ea4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:47.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.164+0000 7f6ea4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:47.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.472+0000 7f6ead98f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:47.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.472+0000 7f6ead98f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:47.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.472+0000 7f6ead98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:47.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.472+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e8c005330 con 0x7f6ea813f650 2024-03-29T10:59:47.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.473+0000 7f6ea4ff9640 1 -- 172.21.15.155:0/186501947 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f6e90050480 con 0x7f6ea813f650 2024-03-29T10:59:47.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.473+0000 7f6ea4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:47.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.473+0000 7f6ea4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:47.455 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:47.455 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.476+0000 7f6ead98f640 10 librados: watch_flush enter 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.476+0000 7f6ead98f640 10 librados: watch_flush exit 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.476+0000 7f6ead98f640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.476+0000 7f6ead98f640 10 client.4553.objecter successfully canceled tick 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.476+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 >> v1:172.21.15.183:6816/1412921548 conn(0x7f6e8806dd40 legacy=0x7f6e88070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 10 monclient: shutdown 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 >> v1:172.21.15.155:6790/0 conn(0x7f6ea813f650 legacy=0x7f6ea8137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 shutdown_connections 2024-03-29T10:59:47.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 >> 172.21.15.155:0/186501947 conn(0x7f6ea8066a00 msgr2=0x7f6ea813b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:47.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 shutdown_connections 2024-03-29T10:59:47.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 1 -- 172.21.15.155:0/186501947 wait complete. 2024-03-29T10:59:47.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:47.477+0000 7f6ead98f640 1 librados: shutdown 2024-03-29T10:59:47.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:54.277 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:54.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.444+0000 7f7e062fb640 10 monclient: get_monmap_and_config 2024-03-29T10:59:54.425 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.444+0000 7f7e062fb640 10 monclient: build_initial_monmap 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.444+0000 7f7e062fb640 10 monclient: monmap: 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:59:54.446552+0000 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:59:54.446552+0000 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T10:59:54.426 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:54.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.445+0000 7f7e062fb640 1 Processor -- start 2024-03-29T10:59:54.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 1 -- start start 2024-03-29T10:59:54.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient: init 2024-03-29T10:59:54.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:54.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:54.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient(hunting): picked mon.noname-b con 0x7f7e00145430 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:54.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient(hunting): picked mon.noname-a con 0x7f7e00145810 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:54.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient(hunting): picked mon.noname-c con 0x7f7e000b32b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:54.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e000b26c0 con 0x7f7e00145810 2024-03-29T10:59:54.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e000aafd0 con 0x7f7e000b32b0 2024-03-29T10:59:54.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e000b3970 con 0x7f7e00145430 2024-03-29T10:59:54.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:54.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.446+0000 7f7e062fb640 10 monclient(hunting): authenticate will time out at 1297.739865s 2024-03-29T10:59:54.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7e05afa640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7e000b32b0 0x7f7e0014d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:56399/0 (socket says 172.21.15.155:56399) 2024-03-29T10:59:54.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7e05afa640 1 -- 172.21.15.155:0/4034692124 learned_addr learned my addr 172.21.15.155:0/4034692124 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:54.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732265009 0 0) 0x7f7e000b3970 con 0x7f7e00145430 2024-03-29T10:59:54.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7df7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:54.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:54.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:54.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7df7fff640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:59:54.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.450+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec003520 con 0x7f7e00145430 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415905036 0 0) 0x7f7e000aafd0 con 0x7f7e000b32b0 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e000b3970 con 0x7f7e000b32b0 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 505311724 0 0) 0x7f7dec003520 con 0x7f7e00145430 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 >> v1:172.21.15.183:6789/0 conn(0x7f7e000b32b0 legacy=0x7f7e0014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 >> v1:172.21.15.155:6789/0 conn(0x7f7e00145810 legacy=0x7f7e000b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 monclient: found mon.noname-b 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T10:59:54.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e000b3b00 con 0x7f7e00145430 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7df0001610 con 0x7f7e00145430 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.453+0000 7f7df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 10 monclient: dump: 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:54.436 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7e062fb640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7e062fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7df0001770 con 0x7f7e00145430 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 1 -- 172.21.15.155:0/4034692124 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7df0001b80 con 0x7f7e00145430 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7df7fff640 10 monclient: dump: 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:54.437 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:54.438 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:54.438 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:54.438 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7e062fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.454+0000 7f7e062fb640 4 set_mon_vals no callback set 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.455+0000 7f7e062fb640 10 monclient: shutdown 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.455+0000 7f7e062fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.455+0000 7f7e062fb640 1 -- 172.21.15.155:0/4034692124 >> v1:172.21.15.155:6790/0 conn(0x7f7e00145430 legacy=0x7f7e000b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.455+0000 7f7e062fb640 1 -- 172.21.15.155:0/4034692124 shutdown_connections 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.455+0000 7f7e062fb640 1 -- 172.21.15.155:0/4034692124 >> 172.21.15.155:0/4034692124 conn(0x7f7e0006d140 msgr2=0x7f7e00066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.455+0000 7f7e062fb640 1 -- 172.21.15.155:0/4034692124 shutdown_connections 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.455+0000 7f7e062fb640 1 -- 172.21.15.155:0/4034692124 wait complete. 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.456+0000 7f7e062fb640 10 monclient: build_initial_monmap 2024-03-29T10:59:54.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.456+0000 7f7e062fb640 10 monclient: monmap: 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.457+0000 7f7e062fb640 1 librados: starting msgr at 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.457+0000 7f7e062fb640 1 librados: starting objecter 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.457+0000 7f7e062fb640 1 Processor -- start 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.457+0000 7f7e062fb640 1 -- start start 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.457+0000 7f7e062fb640 1 librados: setting wanted keys 2024-03-29T10:59:54.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.457+0000 7f7e062fb640 1 librados: calling monclient init 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.457+0000 7f7e062fb640 10 monclient: init 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 10 monclient(hunting): picked mon.noname-b con 0x7f7e000b32b0 addr v1:172.21.15.183:6789/0 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 10 monclient(hunting): picked mon.noname-c con 0x7f7e00145430 addr v1:172.21.15.155:6790/0 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 10 monclient(hunting): picked mon.noname-a con 0x7f7e001dbe00 addr v1:172.21.15.155:6789/0 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0013e9b0 con 0x7f7e001dbe00 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e001dc250 con 0x7f7e000b32b0 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0013ebe0 con 0x7f7e00145430 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:54.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.458+0000 7f7e062fb640 10 monclient(hunting): authenticate will time out at 1297.751155s 2024-03-29T10:59:54.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7e05afa640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7e001dbe00 0x7f7e00139cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:43601/0 (socket says 172.21.15.155:43601) 2024-03-29T10:59:54.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7e05afa640 1 -- 172.21.15.155:0/2101748958 learned_addr learned my addr 172.21.15.155:0/2101748958 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T10:59:54.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 client.?.objecter ms_handle_connect 0x7f7e001dbe00 2024-03-29T10:59:54.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:54.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 client.?.objecter ms_handle_connect 0x7f7e00145430 2024-03-29T10:59:54.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:54.444 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712191754 0 0) 0x7f7e0013ebe0 con 0x7f7e00145430 2024-03-29T10:59:54.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:54.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:54.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:54.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:59:54.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de4004eb0 con 0x7f7e00145430 2024-03-29T10:59:54.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233013780 0 0) 0x7f7e0013e9b0 con 0x7f7e001dbe00 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.461+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0013ebe0 con 0x7f7e001dbe00 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 196954506 0 0) 0x7f7de4004eb0 con 0x7f7e00145430 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e0013e9b0 con 0x7f7e00145430 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3978686215 0 0) 0x7f7e0013ebe0 con 0x7f7e001dbe00 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7de4004eb0 con 0x7f7e001dbe00 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7dfc0018f0 con 0x7f7e00145430 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:54.451 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7df8001b70 con 0x7f7e001dbe00 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:54.452 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 952811240 0 0) 0x7f7e0013e9b0 con 0x7f7e00145430 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.462+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 >> v1:172.21.15.183:6789/0 conn(0x7f7e000b32b0 legacy=0x7f7e0013d5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 >> v1:172.21.15.155:6789/0 conn(0x7f7e001dbe00 legacy=0x7f7e00139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 1 monclient: found mon.c 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0013eec0 con 0x7f7e00145430 2024-03-29T10:59:54.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.464+0000 7f7df5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 10 monclient: _renew_subs 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dfc0035f0 con 0x7f7e00145430 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e001365e0 con 0x7f7e00145430 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 10 monclient: _renew_subs 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0013f4c0 con 0x7f7e00145430 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.465+0000 7f7e062fb640 1 librados: init done 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7e062fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7e062fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7e062fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:54.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0004f730 con 0x7f7e00145430 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7df5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7df77fe640 4 set_mon_vals no callback set 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7dfc003900 con 0x7f7e00145430 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7df5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7df5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.467+0000 7f7df5ffb640 10 monclient: dump: 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T10:59:54.455 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T10:59:54.456 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T10:59:54.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.471+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7dfc0198d0 con 0x7f7e00145430 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.471+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f7dfc083820 con 0x7f7e00145430 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.471+0000 7f7df5ffb640 10 client.4559.objecter ms_dispatch 0x7f7e00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.471+0000 7f7df5ffb640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.471+0000 7f7df5ffb640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.471+0000 7f7df5ffb640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.471+0000 7f7e052f9640 10 monclient: get_auth_request con 0x7f7de406de50 auth_method 0 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.472+0000 7f7df5ffb640 10 client.4559.objecter ms_handle_connect 0x7f7de406de50 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.476+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7dfc04f100 con 0x7f7e00145430 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.476+0000 7f7df5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:54.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.476+0000 7f7df5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:54.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.805+0000 7f7e062fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:54.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.805+0000 7f7e062fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:54.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.805+0000 7f7e062fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T10:59:54.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.805+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e00074d80 con 0x7f7e00145430 2024-03-29T10:59:54.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.806+0000 7f7df5ffb640 1 -- 172.21.15.155:0/2101748958 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f7dfc051030 con 0x7f7e00145430 2024-03-29T10:59:54.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.806+0000 7f7df5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:54.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.806+0000 7f7df5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:54.787 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T10:59:54.787 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:54.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.809+0000 7f7e062fb640 10 librados: watch_flush enter 2024-03-29T10:59:54.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.809+0000 7f7e062fb640 10 librados: watch_flush exit 2024-03-29T10:59:54.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T10:59:54.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 10 client.4559.objecter successfully canceled tick 2024-03-29T10:59:54.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 >> v1:172.21.15.183:6816/1412921548 conn(0x7f7de406de50 legacy=0x7f7de4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:54.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 10 monclient: shutdown 2024-03-29T10:59:54.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:54.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 >> v1:172.21.15.155:6790/0 conn(0x7f7e00145430 legacy=0x7f7e001db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:54.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 shutdown_connections 2024-03-29T10:59:54.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 >> 172.21.15.155:0/2101748958 conn(0x7f7e0006d140 msgr2=0x7f7e00066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:54.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 shutdown_connections 2024-03-29T10:59:54.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.810+0000 7f7e062fb640 1 -- 172.21.15.155:0/2101748958 wait complete. 2024-03-29T10:59:54.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T10:59:54.811+0000 7f7e062fb640 1 librados: shutdown 2024-03-29T10:59:54.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:01.615 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:01.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.784+0000 7f780c115640 10 monclient: get_monmap_and_config 2024-03-29T11:00:01.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.784+0000 7f780c115640 10 monclient: build_initial_monmap 2024-03-29T11:00:01.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.784+0000 7f780c115640 10 monclient: monmap: 2024-03-29T11:00:01.765 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:01.785408+0000 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:01.785408+0000 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:01.766 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:01.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.786+0000 7f780c115640 1 Processor -- start 2024-03-29T11:00:01.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.787+0000 7f780c115640 1 -- start start 2024-03-29T11:00:01.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.787+0000 7f780c115640 10 monclient: init 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 10 monclient(hunting): picked mon.noname-a con 0x7f78040b4100 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 10 monclient(hunting): picked mon.noname-b con 0x7f780413ee90 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 10 monclient(hunting): picked mon.noname-c con 0x7f78040af900 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78040afd10 con 0x7f78040b4100 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78040af6f0 con 0x7f78040af900 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78040aaf60 con 0x7f780413ee90 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:01.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.788+0000 7f780c115640 10 monclient(hunting): authenticate will time out at 1305.080134s 2024-03-29T11:00:01.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f78096b0640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f780413ee90 0x7f78040adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:48054/0 (socket says 172.21.15.155:48054) 2024-03-29T11:00:01.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f78096b0640 1 -- 172.21.15.155:0/110987347 learned_addr learned my addr 172.21.15.155:0/110987347 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:01.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655754836 0 0) 0x7f78040aaf60 con 0x7f780413ee90 2024-03-29T11:00:01.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f7808eaf640 10 monclient(hunting): _negotiate 2024-03-29T11:00:01.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f7808eaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:01.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f7808eaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:01.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f7808eaf640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:00:01.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.792+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f0003520 con 0x7f780413ee90 2024-03-29T11:00:01.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.793+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701711805 0 0) 0x7f77f0003520 con 0x7f780413ee90 2024-03-29T11:00:01.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.793+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 >> v1:172.21.15.183:6789/0 conn(0x7f78040af900 legacy=0x7f780413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:01.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 >> v1:172.21.15.155:6789/0 conn(0x7f78040b4100 legacy=0x7f78040a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:01.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 10 monclient: _finish_hunting 0 2024-03-29T11:00:01.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 1 monclient: found mon.noname-b 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780413e970 con 0x7f780413ee90 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 10 monclient: _finish_auth 0 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 10 monclient: _check_auth_tickets 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f780c115640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f780c115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f77f4001690 con 0x7f780413ee90 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f7808eaf640 10 monclient: dump: 2024-03-29T11:00:01.776 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.795+0000 7f780c115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77f4001860 con 0x7f780413ee90 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f7808eaf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:01.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f780c115640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f780c115640 4 set_mon_vals no callback set 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f7808eaf640 1 -- 172.21.15.155:0/110987347 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f77f4001cb0 con 0x7f780413ee90 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f7808eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f7808eaf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f7808eaf640 10 monclient: dump: 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:01.778 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:01.779 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:01.779 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:01.779 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:01.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f780c115640 10 monclient: shutdown 2024-03-29T11:00:01.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f780c115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:01.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f780c115640 1 -- 172.21.15.155:0/110987347 >> v1:172.21.15.155:6790/0 conn(0x7f780413ee90 legacy=0x7f78040adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:01.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f780c115640 1 -- 172.21.15.155:0/110987347 shutdown_connections 2024-03-29T11:00:01.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.796+0000 7f780c115640 1 -- 172.21.15.155:0/110987347 >> 172.21.15.155:0/110987347 conn(0x7f78040b35e0 msgr2=0x7f78040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:01.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.797+0000 7f780c115640 1 -- 172.21.15.155:0/110987347 shutdown_connections 2024-03-29T11:00:01.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.797+0000 7f780c115640 1 -- 172.21.15.155:0/110987347 wait complete. 2024-03-29T11:00:01.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.797+0000 7f780c115640 10 monclient: build_initial_monmap 2024-03-29T11:00:01.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.797+0000 7f780c115640 10 monclient: monmap: 2024-03-29T11:00:01.781 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:01.781 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:01.781 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.798+0000 7f780c115640 1 librados: starting msgr at 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.798+0000 7f780c115640 1 librados: starting objecter 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.798+0000 7f780c115640 1 Processor -- start 2024-03-29T11:00:01.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.798+0000 7f780c115640 1 -- start start 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.798+0000 7f780c115640 1 librados: setting wanted keys 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.798+0000 7f780c115640 1 librados: calling monclient init 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.798+0000 7f780c115640 10 monclient: init 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 10 monclient(hunting): picked mon.noname-a con 0x7f78040af900 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 10 monclient(hunting): picked mon.noname-b con 0x7f78040b4100 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 10 monclient(hunting): picked mon.noname-c con 0x7f780413ee90 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780412d310 con 0x7f78040af900 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7804137ed0 con 0x7f78040b4100 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78041381b0 con 0x7f780413ee90 2024-03-29T11:00:01.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:01.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.799+0000 7f780c115640 10 monclient(hunting): authenticate will time out at 1305.091273s 2024-03-29T11:00:01.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.801+0000 7f780a6b2640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f780413ee90 0x7f780412cc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:48580/0 (socket says 172.21.15.155:48580) 2024-03-29T11:00:01.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.801+0000 7f780a6b2640 1 -- 172.21.15.155:0/2864531954 learned_addr learned my addr 172.21.15.155:0/2864531954 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.802+0000 7f77fa7fc640 10 client.?.objecter ms_handle_connect 0x7f780413ee90 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.802+0000 7f77fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.802+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245767627 0 0) 0x7f78041381b0 con 0x7f780413ee90 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.802+0000 7f77fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e8004eb0 con 0x7f780413ee90 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 client.?.objecter ms_handle_connect 0x7f78040af900 2024-03-29T11:00:01.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453328394 0 0) 0x7f780412d310 con 0x7f78040af900 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78041381b0 con 0x7f78040af900 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862807719 0 0) 0x7f77e8004eb0 con 0x7f780413ee90 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f780412d310 con 0x7f780413ee90 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f77fc001b70 con 0x7f780413ee90 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.803+0000 7f77fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:01.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 10 monclient(hunting): dump: 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:01.793 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4088035949 0 0) 0x7f78041381b0 con 0x7f78040af900 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77e8004eb0 con 0x7f78040af900 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f78000018a0 con 0x7f78040af900 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 10 monclient(hunting): dump: 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:01.794 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 434583811 0 0) 0x7f780412d310 con 0x7f780413ee90 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.804+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 >> v1:172.21.15.183:6789/0 conn(0x7f78040b4100 legacy=0x7f780412c4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 >> v1:172.21.15.155:6789/0 conn(0x7f78040af900 legacy=0x7f78040b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 1 monclient: found mon.c 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:01.795 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7804138490 con 0x7f780413ee90 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f77fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 10 monclient: _renew_subs 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78041334f0 con 0x7f780413ee90 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 10 monclient: _renew_subs 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7804132fd0 con 0x7f780413ee90 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.805+0000 7f780c115640 1 librados: init done 2024-03-29T11:00:01.796 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.806+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77fc001cd0 con 0x7f780413ee90 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.806+0000 7f77fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.806+0000 7f77fbfff640 4 set_mon_vals no callback set 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.807+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f77fc002180 con 0x7f780413ee90 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.807+0000 7f77fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.807+0000 7f77fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.807+0000 7f77fa7fc640 10 monclient: dump: 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:01.797 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.808+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f77fc01ac90 con 0x7f780413ee90 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.808+0000 7f780c115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.808+0000 7f780c115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.808+0000 7f780c115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.808+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec005080 con 0x7f780413ee90 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.809+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f77fc084ce0 con 0x7f780413ee90 2024-03-29T11:00:01.798 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.809+0000 7f77fa7fc640 10 client.4565.objecter ms_dispatch 0x7f7804000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.809+0000 7f77fa7fc640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.809+0000 7f77fa7fc640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.809+0000 7f7809eb1640 10 monclient: get_auth_request con 0x7f77e806de80 auth_method 0 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.809+0000 7f77fa7fc640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.810+0000 7f77fa7fc640 10 client.4565.objecter ms_handle_connect 0x7f77e806de80 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.817+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77fc0505c0 con 0x7f780413ee90 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.817+0000 7f77fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:01.799 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:01.817+0000 7f77fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:02.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.142+0000 7f780c115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:02.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.142+0000 7f780c115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:02.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.142+0000 7f780c115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:02.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.142+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77ec005330 con 0x7f780413ee90 2024-03-29T11:00:02.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.145+0000 7f77fa7fc640 1 -- 172.21.15.155:0/2864531954 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f77fc0524f0 con 0x7f780413ee90 2024-03-29T11:00:02.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.145+0000 7f77fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:02.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.145+0000 7f77fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:02.126 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:02.126 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:02.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.149+0000 7f780c115640 10 librados: watch_flush enter 2024-03-29T11:00:02.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 10 librados: watch_flush exit 2024-03-29T11:00:02.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 >> v1:172.21.15.183:6816/1412921548 conn(0x7f77e806de80 legacy=0x7f77e8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 10 monclient: shutdown 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 >> v1:172.21.15.155:6790/0 conn(0x7f780413ee90 legacy=0x7f780412cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 shutdown_connections 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 >> 172.21.15.155:0/2864531954 conn(0x7f78040b35e0 msgr2=0x7f78041370a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.150+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 shutdown_connections 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.151+0000 7f780c115640 1 -- 172.21.15.155:0/2864531954 wait complete. 2024-03-29T11:00:02.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:02.151+0000 7f780c115640 1 librados: shutdown 2024-03-29T11:00:02.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:08.954 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.127+0000 7f7c04cf1640 10 monclient: get_monmap_and_config 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.127+0000 7f7c04cf1640 10 monclient: build_initial_monmap 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.127+0000 7f7c04cf1640 10 monclient: monmap: 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:09.128457+0000 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:09.128457+0000 2024-03-29T11:00:09.108 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:09.109 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:09.109 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:09.109 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:09.109 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:09.109 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:09.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.131+0000 7f7c04cf1640 1 Processor -- start 2024-03-29T11:00:09.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.131+0000 7f7c04cf1640 1 -- start start 2024-03-29T11:00:09.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.131+0000 7f7c04cf1640 10 monclient: init 2024-03-29T11:00:09.112 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.132+0000 7f7c04cf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.132+0000 7f7c04cf1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.132+0000 7f7c04cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.132+0000 7f7c04cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c0013ef30 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.133+0000 7f7c04cf1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c000af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.133+0000 7f7c04cf1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000b0c40 con 0x7f7c000b0860 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.133+0000 7f7c04cf1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000aed10 con 0x7f7c0013ef30 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.133+0000 7f7c04cf1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000afd50 con 0x7f7c000af8e0 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.133+0000 7f7c04cf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:09.113 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.133+0000 7f7c04cf1640 10 monclient(hunting): authenticate will time out at 1312.425005s 2024-03-29T11:00:09.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.135+0000 7f7bfe59c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7c000b0860 0x7f7c0013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:33319/0 (socket says 172.21.15.155:33319) 2024-03-29T11:00:09.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.135+0000 7f7bfe59c640 1 -- 172.21.15.155:0/2622852438 learned_addr learned my addr 172.21.15.155:0/2622852438 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:09.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.135+0000 7f7bfed9d640 1 --1- 172.21.15.155:0/2622852438 >> v1:172.21.15.155:6790/0 conn(0x7f7c000af8e0 0x7f7c00149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:55406/0 (socket says 172.21.15.155:55406) 2024-03-29T11:00:09.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.138+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363317152 0 0) 0x7f7c000afd50 con 0x7f7c000af8e0 2024-03-29T11:00:09.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:09.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:09.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:09.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:00:09.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be8003520 con 0x7f7c000af8e0 2024-03-29T11:00:09.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927673407 0 0) 0x7f7c000b0c40 con 0x7f7c000b0860 2024-03-29T11:00:09.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:09.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:09.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:09.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:00:09.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.139+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c000afd50 con 0x7f7c000b0860 2024-03-29T11:00:09.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543908333 0 0) 0x7f7be8003520 con 0x7f7c000af8e0 2024-03-29T11:00:09.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 >> v1:172.21.15.183:6789/0 conn(0x7f7c0013ef30 legacy=0x7f7c000af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 >> v1:172.21.15.155:6789/0 conn(0x7f7c000b0860 legacy=0x7f7c0013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:00:09.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 1 monclient: found mon.noname-b 2024-03-29T11:00:09.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.140+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000ab1b0 con 0x7f7c000af8e0 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: _finish_auth 0 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7bf00015c0 con 0x7f7c000af8e0 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: dump: 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.122 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bf00017e0 con 0x7f7c000af8e0 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 1 -- 172.21.15.155:0/2622852438 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7bf0001b70 con 0x7f7c000af8e0 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.141+0000 7f7bfd59a640 10 monclient: dump: 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:09.123 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.124 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.124 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.124 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:09.124 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:09.124 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:09.124 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:09.124 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.142+0000 7f7c04cf1640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.142+0000 7f7c04cf1640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.142+0000 7f7c04cf1640 4 set_mon_vals no callback set 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.143+0000 7f7c04cf1640 10 monclient: shutdown 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.143+0000 7f7c04cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.143+0000 7f7c04cf1640 1 -- 172.21.15.155:0/2622852438 >> v1:172.21.15.155:6790/0 conn(0x7f7c000af8e0 legacy=0x7f7c00149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.145+0000 7f7c04cf1640 1 -- 172.21.15.155:0/2622852438 shutdown_connections 2024-03-29T11:00:09.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.145+0000 7f7c04cf1640 1 -- 172.21.15.155:0/2622852438 >> 172.21.15.155:0/2622852438 conn(0x7f7c0006b660 msgr2=0x7f7c0006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:09.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.145+0000 7f7c04cf1640 1 -- 172.21.15.155:0/2622852438 shutdown_connections 2024-03-29T11:00:09.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.146+0000 7f7c04cf1640 1 -- 172.21.15.155:0/2622852438 wait complete. 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.147+0000 7f7c04cf1640 10 monclient: build_initial_monmap 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.147+0000 7f7c04cf1640 10 monclient: monmap: 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:09.127 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:09.128 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:09.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.148+0000 7f7c04cf1640 1 librados: starting msgr at 2024-03-29T11:00:09.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.148+0000 7f7c04cf1640 1 librados: starting objecter 2024-03-29T11:00:09.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.148+0000 7f7c04cf1640 1 Processor -- start 2024-03-29T11:00:09.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.148+0000 7f7c04cf1640 1 -- start start 2024-03-29T11:00:09.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.148+0000 7f7c04cf1640 1 librados: setting wanted keys 2024-03-29T11:00:09.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.148+0000 7f7c04cf1640 1 librados: calling monclient init 2024-03-29T11:00:09.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.148+0000 7f7c04cf1640 10 monclient: init 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.149+0000 7f7c04cf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.149+0000 7f7c04cf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.149+0000 7f7c04cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000af8e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.149+0000 7f7c04cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c000b0860 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.149+0000 7f7c04cf1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0013ef30 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.149+0000 7f7c04cf1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000b4250 con 0x7f7c000af8e0 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.150+0000 7f7c04cf1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c00135170 con 0x7f7c0013ef30 2024-03-29T11:00:09.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.150+0000 7f7c04cf1640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c001c50d0 con 0x7f7c000b0860 2024-03-29T11:00:09.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.150+0000 7f7c04cf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:09.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.150+0000 7f7c04cf1640 10 monclient(hunting): authenticate will time out at 1312.441901s 2024-03-29T11:00:09.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7bfdd9b640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f7c000b0860 0x7f7c00134350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:54948/0 (socket says 172.21.15.155:54948) 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7bfdd9b640 1 -- 172.21.15.155:0/4142421203 learned_addr learned my addr 172.21.15.155:0/4142421203 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 10 client.?.objecter ms_handle_connect 0x7f7c000b0860 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732795129 0 0) 0x7f7c001c50d0 con 0x7f7c000b0860 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.152+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be0004eb0 con 0x7f7c000b0860 2024-03-29T11:00:09.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.153+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1039356218 0 0) 0x7f7be0004eb0 con 0x7f7c000b0860 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.153+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c001c50d0 con 0x7f7c000b0860 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.153+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7bf4001990 con 0x7f7c000b0860 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.153+0000 7f7beeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.153+0000 7f7beeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.153+0000 7f7beeffd640 10 monclient(hunting): dump: 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:09.136 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:09.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.154+0000 7f7beeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:09.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3277453099 0 0) 0x7f7c001c50d0 con 0x7f7c000b0860 2024-03-29T11:00:09.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 >> v1:172.21.15.183:6789/0 conn(0x7f7c0013ef30 legacy=0x7f7c00134a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:00:09.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 >> v1:172.21.15.155:6789/0 conn(0x7f7c000af8e0 legacy=0x7f7c000b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:09.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 1 monclient: found mon.c 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c001c53b0 con 0x7f7c000b0860 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 10 monclient: _renew_subs 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c000b3730 con 0x7f7c000b0860 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bf4001c90 con 0x7f7c000b0860 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:09.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7bf4003a60 con 0x7f7c000b0860 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beeffd640 10 monclient: dump: 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:09.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7beffff640 4 set_mon_vals no callback set 2024-03-29T11:00:09.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 10 monclient: _renew_subs 2024-03-29T11:00:09.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:09.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c001d6fe0 con 0x7f7c000b0860 2024-03-29T11:00:09.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.156+0000 7f7c04cf1640 1 librados: init done 2024-03-29T11:00:09.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.158+0000 7f7c04cf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:09.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.158+0000 7f7c04cf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:09.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.158+0000 7f7c04cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:09.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.158+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c00075f10 con 0x7f7c000b0860 2024-03-29T11:00:09.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.159+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7bf4003bc0 con 0x7f7c000b0860 2024-03-29T11:00:09.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.159+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f7bf4084af0 con 0x7f7c000b0860 2024-03-29T11:00:09.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.159+0000 7f7beeffd640 10 client.4571.objecter ms_dispatch 0x7f7c00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:09.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.159+0000 7f7beeffd640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:09.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.159+0000 7f7beeffd640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:09.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.159+0000 7f7beeffd640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:00:09.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.162+0000 7f7bfe59c640 10 monclient: get_auth_request con 0x7f7be006dcd0 auth_method 0 2024-03-29T11:00:09.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.163+0000 7f7beeffd640 10 client.4571.objecter ms_handle_connect 0x7f7be006dcd0 2024-03-29T11:00:09.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.171+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7bf40503d0 con 0x7f7c000b0860 2024-03-29T11:00:09.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.171+0000 7f7beeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:09.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.171+0000 7f7beeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.492+0000 7f7c04cf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.492+0000 7f7c04cf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.492+0000 7f7c04cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.492+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c00072330 con 0x7f7c000b0860 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.493+0000 7f7beeffd640 1 -- 172.21.15.155:0/4142421203 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f7bf4052300 con 0x7f7c000b0860 2024-03-29T11:00:09.474 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.493+0000 7f7beeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:09.475 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.493+0000 7f7beeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:09.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.497+0000 7f7c04cf1640 10 librados: watch_flush enter 2024-03-29T11:00:09.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.497+0000 7f7c04cf1640 10 librados: watch_flush exit 2024-03-29T11:00:09.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.497+0000 7f7c04cf1640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:00:09.477 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.497+0000 7f7c04cf1640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 >> v1:172.21.15.183:6816/1412921548 conn(0x7f7be006dcd0 legacy=0x7f7be0070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 10 monclient: shutdown 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 >> v1:172.21.15.155:6790/0 conn(0x7f7c000b0860 legacy=0x7f7c00134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 shutdown_connections 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 >> 172.21.15.155:0/4142421203 conn(0x7f7c0006b660 msgr2=0x7f7c00069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 shutdown_connections 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 1 -- 172.21.15.155:0/4142421203 wait complete. 2024-03-29T11:00:09.478 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:09.498+0000 7f7c04cf1640 1 librados: shutdown 2024-03-29T11:00:09.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:16.301 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.483+0000 7fe937988640 10 monclient: get_monmap_and_config 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.483+0000 7fe937988640 10 monclient: build_initial_monmap 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.483+0000 7fe937988640 10 monclient: monmap: 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:16.483786+0000 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:16.483786+0000 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:16.463 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:16.464 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:16.464 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.484+0000 7fe937988640 1 Processor -- start 2024-03-29T11:00:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.484+0000 7fe937988640 1 -- start start 2024-03-29T11:00:16.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.484+0000 7fe937988640 10 monclient: init 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 10 monclient(hunting): picked mon.noname-a con 0x7fe930149ed0 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 10 monclient(hunting): picked mon.noname-b con 0x7fe930072370 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 10 monclient(hunting): picked mon.noname-c con 0x7fe93014a2b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93014aa50 con 0x7fe930149ed0 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93014a760 con 0x7fe93014a2b0 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe930070190 con 0x7fe930072370 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:16.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.485+0000 7fe937988640 10 monclient(hunting): authenticate will time out at 1319.777678s 2024-03-29T11:00:16.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.492+0000 7fe935f25640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe93014a2b0 0x7fe93006c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:52753/0 (socket says 172.21.15.155:52753) 2024-03-29T11:00:16.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.492+0000 7fe935724640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe930149ed0 0x7fe930071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36747/0 (socket says 172.21.15.155:36747) 2024-03-29T11:00:16.472 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.492+0000 7fe935f25640 1 -- 172.21.15.155:0/2781434875 learned_addr learned my addr 172.21.15.155:0/2781434875 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.498+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240897798 0 0) 0x7fe93014aa50 con 0x7fe930149ed0 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.498+0000 7fe927fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.498+0000 7fe927fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.498+0000 7fe927fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.498+0000 7fe927fff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.498+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe91c003520 con 0x7fe930149ed0 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.499+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604257219 0 0) 0x7fe93014a760 con 0x7fe93014a2b0 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.499+0000 7fe927fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.499+0000 7fe927fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.499+0000 7fe927fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.499+0000 7fe927fff640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:00:16.479 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.499+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe93014aa50 con 0x7fe93014a2b0 2024-03-29T11:00:16.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000937495 0 0) 0x7fe93014aa50 con 0x7fe93014a2b0 2024-03-29T11:00:16.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 >> v1:172.21.15.155:6790/0 conn(0x7fe930072370 legacy=0x7fe93006c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:16.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 >> v1:172.21.15.155:6789/0 conn(0x7fe930149ed0 legacy=0x7fe930071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:16.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:16.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 monclient: found mon.noname-c 2024-03-29T11:00:16.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:16.481 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930070470 con 0x7fe93014a2b0 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe9200015f0 con 0x7fe93014a2b0 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: dump: 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.482 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe920001750 con 0x7fe93014a2b0 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 1 -- 172.21.15.155:0/2781434875 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe920001ae0 con 0x7fe93014a2b0 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.501+0000 7fe927fff640 10 monclient: dump: 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.483 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:16.484 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:16.484 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:16.484 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:16.484 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:16.484 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:16.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.503+0000 7fe937988640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:00:16.484 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.503+0000 7fe937988640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:16.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.504+0000 7fe937988640 4 set_mon_vals no callback set 2024-03-29T11:00:16.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.505+0000 7fe937988640 10 monclient: shutdown 2024-03-29T11:00:16.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.505+0000 7fe937988640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:16.485 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.506+0000 7fe937988640 1 -- 172.21.15.155:0/2781434875 >> v1:172.21.15.183:6789/0 conn(0x7fe93014a2b0 legacy=0x7fe93006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:16.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.507+0000 7fe937988640 1 -- 172.21.15.155:0/2781434875 shutdown_connections 2024-03-29T11:00:16.486 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.507+0000 7fe937988640 1 -- 172.21.15.155:0/2781434875 >> 172.21.15.155:0/2781434875 conn(0x7fe9300675b0 msgr2=0x7fe9300b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:16.487 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.507+0000 7fe937988640 1 -- 172.21.15.155:0/2781434875 shutdown_connections 2024-03-29T11:00:16.490 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.510+0000 7fe937988640 1 -- 172.21.15.155:0/2781434875 wait complete. 2024-03-29T11:00:16.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.512+0000 7fe937988640 10 monclient: build_initial_monmap 2024-03-29T11:00:16.491 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.512+0000 7fe937988640 10 monclient: monmap: 2024-03-29T11:00:16.491 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.512+0000 7fe937988640 1 librados: starting msgr at 2024-03-29T11:00:16.492 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.512+0000 7fe937988640 1 librados: starting objecter 2024-03-29T11:00:16.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.513+0000 7fe937988640 1 Processor -- start 2024-03-29T11:00:16.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.513+0000 7fe937988640 1 -- start start 2024-03-29T11:00:16.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.513+0000 7fe937988640 1 librados: setting wanted keys 2024-03-29T11:00:16.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.513+0000 7fe937988640 1 librados: calling monclient init 2024-03-29T11:00:16.493 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.513+0000 7fe937988640 10 monclient: init 2024-03-29T11:00:16.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:16.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:16.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 10 monclient(hunting): picked mon.noname-a con 0x7fe930072370 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:16.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 10 monclient(hunting): picked mon.noname-c con 0x7fe930149ed0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:16.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 10 monclient(hunting): picked mon.noname-b con 0x7fe93014a2b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:16.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe930121860 con 0x7fe930072370 2024-03-29T11:00:16.494 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9300abde0 con 0x7fe93014a2b0 2024-03-29T11:00:16.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9300ac050 con 0x7fe930149ed0 2024-03-29T11:00:16.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:16.495 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.514+0000 7fe937988640 10 monclient(hunting): authenticate will time out at 1319.806705s 2024-03-29T11:00:16.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.520+0000 7fe935724640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe930072370 0x7fe9300ae310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:37249/0 (socket says 172.21.15.155:37249) 2024-03-29T11:00:16.499 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.520+0000 7fe935724640 1 -- 172.21.15.155:0/2862217210 learned_addr learned my addr 172.21.15.155:0/2862217210 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:16.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 client.?.objecter ms_handle_connect 0x7fe930072370 2024-03-29T11:00:16.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:16.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 client.?.objecter ms_handle_connect 0x7fe93014a2b0 2024-03-29T11:00:16.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:16.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904559603 0 0) 0x7fe9300abde0 con 0x7fe93014a2b0 2024-03-29T11:00:16.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:16.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe914004eb0 con 0x7fe93014a2b0 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773922128 0 0) 0x7fe930121860 con 0x7fe930072370 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9300abde0 con 0x7fe930072370 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 975568515 0 0) 0x7fe914004eb0 con 0x7fe93014a2b0 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe930121860 con 0x7fe93014a2b0 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe920001a60 con 0x7fe93014a2b0 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:16.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.524+0000 7fe925ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3533334099 0 0) 0x7fe930121860 con 0x7fe93014a2b0 2024-03-29T11:00:16.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 >> v1:172.21.15.155:6790/0 conn(0x7fe930149ed0 legacy=0x7fe9300aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 >> v1:172.21.15.155:6789/0 conn(0x7fe930072370 legacy=0x7fe9300ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 1 monclient: found mon.b 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9300ac330 con 0x7fe93014a2b0 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.525+0000 7fe925ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe920001ce0 con 0x7fe93014a2b0 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe925ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe920004c30 con 0x7fe93014a2b0 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe925ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:16.511 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe925ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe925ffb640 10 monclient: dump: 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe926ffd640 4 set_mon_vals no callback set 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe937988640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:00:16.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe937988640 10 monclient: _renew_subs 2024-03-29T11:00:16.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe937988640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:16.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.526+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93011d710 con 0x7fe93014a2b0 2024-03-29T11:00:16.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.527+0000 7fe937988640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:16.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.527+0000 7fe937988640 10 monclient: _renew_subs 2024-03-29T11:00:16.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.527+0000 7fe937988640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:16.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.527+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe930121480 con 0x7fe93014a2b0 2024-03-29T11:00:16.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.527+0000 7fe937988640 1 librados: init done 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.529+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fe92001ac00 con 0x7fe93014a2b0 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.530+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fe9200849c0 con 0x7fe93014a2b0 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.530+0000 7fe925ffb640 10 client.4486.objecter ms_dispatch 0x7fe930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.530+0000 7fe925ffb640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.530+0000 7fe925ffb640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.530+0000 7fe925ffb640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.530+0000 7fe935724640 10 monclient: get_auth_request con 0x7fe91406dcb0 auth_method 0 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.531+0000 7fe925ffb640 10 client.4486.objecter ms_handle_connect 0x7fe91406dcb0 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.532+0000 7fe937988640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.533+0000 7fe937988640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:16.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.533+0000 7fe937988640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:16.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.533+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9300af310 con 0x7fe93014a2b0 2024-03-29T11:00:16.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.541+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9200502a0 con 0x7fe93014a2b0 2024-03-29T11:00:16.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.541+0000 7fe925ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:16.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.541+0000 7fe925ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:16.883 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.901+0000 7fe937988640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.901+0000 7fe937988640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.901+0000 7fe937988640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.901+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe930075f10 con 0x7fe93014a2b0 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.902+0000 7fe925ffb640 1 -- 172.21.15.155:0/2862217210 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fe9200521d0 con 0x7fe93014a2b0 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.902+0000 7fe925ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:16.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.902+0000 7fe925ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:16.889 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.909+0000 7fe937988640 10 librados: watch_flush enter 2024-03-29T11:00:16.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.912+0000 7fe937988640 10 librados: watch_flush exit 2024-03-29T11:00:16.892 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.912+0000 7fe937988640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.912+0000 7fe937988640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.912+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 >> v1:172.21.15.183:6816/1412921548 conn(0x7fe91406dcb0 legacy=0x7fe914070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.912+0000 7fe937988640 10 monclient: shutdown 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.913+0000 7fe937988640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.913+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 >> v1:172.21.15.183:6789/0 conn(0x7fe93014a2b0 legacy=0x7fe9301e6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.913+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 shutdown_connections 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.913+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 >> 172.21.15.155:0/2862217210 conn(0x7fe9300675b0 msgr2=0x7fe9300b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:16.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.913+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 shutdown_connections 2024-03-29T11:00:16.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.913+0000 7fe937988640 1 -- 172.21.15.155:0/2862217210 wait complete. 2024-03-29T11:00:16.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:16.914+0000 7fe937988640 1 librados: shutdown 2024-03-29T11:00:16.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:23.721 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:23.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.889+0000 7f4662a0d640 10 monclient: get_monmap_and_config 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.889+0000 7f4662a0d640 10 monclient: build_initial_monmap 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.889+0000 7f4662a0d640 10 monclient: monmap: 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:23.890435+0000 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:23.890435+0000 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:23.870 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:23.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.891+0000 7f4662a0d640 1 Processor -- start 2024-03-29T11:00:23.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.891+0000 7f4662a0d640 1 -- start start 2024-03-29T11:00:23.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.891+0000 7f4662a0d640 10 monclient: init 2024-03-29T11:00:23.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.892+0000 7f4662a0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:23.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.892+0000 7f4662a0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:23.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.892+0000 7f4662a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f465c0b0770 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:23.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4662a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f465c14a700 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:23.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4662a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f465c0ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:23.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4662a0d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c0aba70 con 0x7f465c0b0770 2024-03-29T11:00:23.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4662a0d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c0aecb0 con 0x7f465c14a700 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4660faa640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f465c0ac5a0 0x7f465c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44564/0 (socket says 172.21.15.155:44564) 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4660faa640 1 -- 172.21.15.155:0/2439696453 learned_addr learned my addr 172.21.15.155:0/2439696453 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4662a0d640 1 -- 172.21.15.155:0/2439696453 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c0af150 con 0x7f465c0ac5a0 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4662a0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f4662a0d640 10 monclient(hunting): authenticate will time out at 1327.185576s 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158695396 0 0) 0x7f465c0af150 con 0x7f465c0ac5a0 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f465affd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f465affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f465affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f465affd640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.893+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4644003520 con 0x7f465c0ac5a0 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204022303 0 0) 0x7f4644003520 con 0x7f465c0ac5a0 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 >> v1:172.21.15.183:6789/0 conn(0x7f465c14a700 legacy=0x7f465c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 >> v1:172.21.15.155:6789/0 conn(0x7f465c0b0770 legacy=0x7f465c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:23.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 1 monclient: found mon.noname-b 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c0ac9d0 con 0x7f465c0ac5a0 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 10 monclient: _finish_auth 0 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4654001a10 con 0x7f465c0ac5a0 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f465affd640 10 monclient: dump: 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.881 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f4662a0d640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.896+0000 7f4662a0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4654001ca0 con 0x7f465c0ac5a0 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f465affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f465affd640 1 -- 172.21.15.155:0/2439696453 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4654002030 con 0x7f465c0ac5a0 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f465affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f465affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:23.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f465affd640 10 monclient: dump: 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f4662a0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f4662a0d640 4 set_mon_vals no callback set 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f4662a0d640 10 monclient: shutdown 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f4662a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:23.883 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.897+0000 7f4662a0d640 1 -- 172.21.15.155:0/2439696453 >> v1:172.21.15.155:6790/0 conn(0x7f465c0ac5a0 legacy=0x7f465c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:23.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.898+0000 7f4662a0d640 1 -- 172.21.15.155:0/2439696453 shutdown_connections 2024-03-29T11:00:23.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.898+0000 7f4662a0d640 1 -- 172.21.15.155:0/2439696453 >> 172.21.15.155:0/2439696453 conn(0x7f465c06b660 msgr2=0x7f465c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:23.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.898+0000 7f4662a0d640 1 -- 172.21.15.155:0/2439696453 shutdown_connections 2024-03-29T11:00:23.884 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.898+0000 7f4662a0d640 1 -- 172.21.15.155:0/2439696453 wait complete. 2024-03-29T11:00:23.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 10 monclient: build_initial_monmap 2024-03-29T11:00:23.885 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 10 monclient: monmap: 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:23.886 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 1 librados: starting msgr at 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 1 librados: starting objecter 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 1 Processor -- start 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 1 -- start start 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 1 librados: setting wanted keys 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 1 librados: calling monclient init 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.899+0000 7f4662a0d640 10 monclient: init 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.900+0000 7f4662a0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.900+0000 7f4662a0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.900+0000 7f4662a0d640 10 monclient(hunting): picked mon.noname-c con 0x7f465c0ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.900+0000 7f465bfff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f465c0ac5a0 0x7f465c122ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:47002/0 (socket says 172.21.15.155:47002) 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.904+0000 7f465bfff640 1 -- 172.21.15.155:0/2347339017 learned_addr learned my addr 172.21.15.155:0/2347339017 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:23.887 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.904+0000 7f4662a0d640 10 monclient(hunting): picked mon.noname-b con 0x7f465c0b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:23.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.905+0000 7f4658ff9640 10 client.?.objecter ms_handle_connect 0x7f465c0ac5a0 2024-03-29T11:00:23.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.905+0000 7f4658ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:23.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.905+0000 7f4658ff9640 10 client.?.objecter ms_handle_connect 0x7f465c0b0770 2024-03-29T11:00:23.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.905+0000 7f4658ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:23.893 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.906+0000 7f4662a0d640 10 monclient(hunting): picked mon.noname-a con 0x7f465c14a700 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.906+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c11ffe0 con 0x7f465c14a700 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.906+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c1231d0 con 0x7f465c0b0770 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.906+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c123360 con 0x7f465c0ac5a0 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.906+0000 7f4662a0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.906+0000 7f4662a0d640 10 monclient(hunting): authenticate will time out at 1327.198566s 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492337923 0 0) 0x7f465c1231d0 con 0x7f465c0b0770 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4648004eb0 con 0x7f465c0b0770 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855924263 0 0) 0x7f465c123360 con 0x7f465c0ac5a0 2024-03-29T11:00:23.894 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.907+0000 7f4658ff9640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c1231d0 con 0x7f465c0ac5a0 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 client.?.objecter ms_handle_connect 0x7f465c14a700 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 198930179 0 0) 0x7f4648004eb0 con 0x7f465c0b0770 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f465c123360 con 0x7f465c0b0770 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666451295 0 0) 0x7f465c11ffe0 con 0x7f465c14a700 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:00:23.895 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4648004eb0 con 0x7f465c14a700 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4650001700 con 0x7f465c0b0770 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:23.896 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1193554179 0 0) 0x7f465c1231d0 con 0x7f465c0ac5a0 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.908+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f465c11ffe0 con 0x7f465c0ac5a0 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f464c001970 con 0x7f465c0ac5a0 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:23.897 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 680648832 0 0) 0x7f465c123360 con 0x7f465c0b0770 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 >> v1:172.21.15.155:6790/0 conn(0x7f465c0ac5a0 legacy=0x7f465c122ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 >> v1:172.21.15.155:6789/0 conn(0x7f465c14a700 legacy=0x7f465c11e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 1 monclient: found mon.b 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.909+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c1234f0 con 0x7f465c0b0770 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4650001860 con 0x7f465c0b0770 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:23.898 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f46500035f0 con 0x7f465c0b0770 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4658ff9640 10 monclient: dump: 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 10 monclient: _renew_subs 2024-03-29T11:00:23.899 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c1e46a0 con 0x7f465c0b0770 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f465a7fc640 4 set_mon_vals no callback set 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 10 monclient: _renew_subs 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f465c11e730 con 0x7f465c0b0770 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.910+0000 7f4662a0d640 1 librados: init done 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.913+0000 7f4662a0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.913+0000 7f4662a0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.913+0000 7f4662a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:23.900 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.913+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c11dc60 con 0x7f465c0b0770 2024-03-29T11:00:23.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.913+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f46500195c0 con 0x7f465c0b0770 2024-03-29T11:00:23.901 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.914+0000 7f465bfff640 10 monclient: get_auth_request con 0x7f464806def0 auth_method 0 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.914+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f46500835c0 con 0x7f465c0b0770 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.914+0000 7f4658ff9640 10 client.4489.objecter ms_dispatch 0x7f465c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.914+0000 7f4658ff9640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.914+0000 7f4658ff9640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.914+0000 7f4658ff9640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.915+0000 7f4658ff9640 10 client.4489.objecter ms_handle_connect 0x7f464806def0 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.922+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f465004ecd0 con 0x7f465c0b0770 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.922+0000 7f4658ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:23.902 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:23.922+0000 7f4658ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.254+0000 7f4662a0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.254+0000 7f4662a0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.254+0000 7f4662a0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:24.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.254+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f465c075f10 con 0x7f465c0b0770 2024-03-29T11:00:24.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.256+0000 7f4658ff9640 1 -- 172.21.15.155:0/2347339017 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f4650050dd0 con 0x7f465c0b0770 2024-03-29T11:00:24.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.257+0000 7f4658ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:24.238 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.257+0000 7f4658ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:24.239 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:24.239 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 10 librados: watch_flush enter 2024-03-29T11:00:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 10 librados: watch_flush exit 2024-03-29T11:00:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-29T11:00:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 10 client.4489.objecter successfully canceled tick 2024-03-29T11:00:24.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 >> v1:172.21.15.183:6816/1412921548 conn(0x7f464806def0 legacy=0x7f46480703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 10 monclient: shutdown 2024-03-29T11:00:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.263+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 >> v1:172.21.15.183:6789/0 conn(0x7f465c0b0770 legacy=0x7f465c1e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.264+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 shutdown_connections 2024-03-29T11:00:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.264+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 >> 172.21.15.155:0/2347339017 conn(0x7f465c06b660 msgr2=0x7f465c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:24.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.264+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 shutdown_connections 2024-03-29T11:00:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.264+0000 7f4662a0d640 1 -- 172.21.15.155:0/2347339017 wait complete. 2024-03-29T11:00:24.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:24.265+0000 7f4662a0d640 1 librados: shutdown 2024-03-29T11:00:24.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:31.070 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:31.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.235+0000 7f5ec1fa3640 10 monclient: get_monmap_and_config 2024-03-29T11:00:31.217 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.235+0000 7f5ec1fa3640 10 monclient: build_initial_monmap 2024-03-29T11:00:31.218 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.235+0000 7f5ec1fa3640 10 monclient: monmap: 2024-03-29T11:00:31.218 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:31.218 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:31.218 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:31.236968+0000 2024-03-29T11:00:31.219 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:31.236968+0000 2024-03-29T11:00:31.219 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:31.219 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.219 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:31.219 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:31.220 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:31.220 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.236+0000 7f5ec1fa3640 1 Processor -- start 2024-03-29T11:00:31.244 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.236+0000 7f5ec1fa3640 1 -- start start 2024-03-29T11:00:31.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.236+0000 7f5ec1fa3640 10 monclient: init 2024-03-29T11:00:31.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:31.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:31.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f5ebc0b0860 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:31.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f5ebc13ef30 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:31.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f5ebc0af8e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:31.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0b0c40 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0aed10 con 0x7f5ebc0b0860 2024-03-29T11:00:31.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc0afd50 con 0x7f5ebc13ef30 2024-03-29T11:00:31.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:31.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.237+0000 7f5ec1fa3640 10 monclient(hunting): authenticate will time out at 1334.530248s 2024-03-29T11:00:31.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5ebb7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f5ebc0b0860 0x7f5ebc13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:39539/0 (socket says 172.21.15.155:39539) 2024-03-29T11:00:31.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5ebb7fe640 1 -- 172.21.15.155:0/1772735348 learned_addr learned my addr 172.21.15.155:0/1772735348 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:31.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574596279 0 0) 0x7f5ebc0aed10 con 0x7f5ebc0b0860 2024-03-29T11:00:31.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5eba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5eba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5eba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5eba7fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:00:31.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.240+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea4003520 con 0x7f5ebc0b0860 2024-03-29T11:00:31.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731936159 0 0) 0x7f5ebc0b0c40 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:00:31.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc0aed10 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 688729943 0 0) 0x7f5ea4003520 con 0x7f5ebc0b0860 2024-03-29T11:00:31.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 >> v1:172.21.15.155:6790/0 conn(0x7f5ebc13ef30 legacy=0x7f5ebc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:31.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 >> v1:172.21.15.155:6789/0 conn(0x7f5ebc0af8e0 legacy=0x7f5ebc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:31.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 1 monclient: found mon.noname-c 2024-03-29T11:00:31.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:31.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:00:31.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.241+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0ab1b0 con 0x7f5ebc0b0860 2024-03-29T11:00:31.250 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5eb4001a10 con 0x7f5ebc0b0860 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: dump: 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:31.251 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5ec1fa3640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5ec1fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb4001c60 con 0x7f5ebc0b0860 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 1 -- 172.21.15.155:0/1772735348 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5eb4001ff0 con 0x7f5ebc0b0860 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5ec1fa3640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5ec1fa3640 4 set_mon_vals no callback set 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.242+0000 7f5eba7fc640 10 monclient: dump: 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.252 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.243+0000 7f5ec1fa3640 10 monclient: shutdown 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.243+0000 7f5ec1fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.243+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/1772735348 >> v1:172.21.15.183:6789/0 conn(0x7f5ebc0b0860 legacy=0x7f5ebc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.243+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/1772735348 shutdown_connections 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.243+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/1772735348 >> 172.21.15.155:0/1772735348 conn(0x7f5ebc06b660 msgr2=0x7f5ebc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.244+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/1772735348 shutdown_connections 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.244+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/1772735348 wait complete. 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.246+0000 7f5ec1fa3640 10 monclient: build_initial_monmap 2024-03-29T11:00:31.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.246+0000 7f5ec1fa3640 10 monclient: monmap: 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.246+0000 7f5ec1fa3640 1 librados: starting msgr at 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.246+0000 7f5ec1fa3640 1 librados: starting objecter 2024-03-29T11:00:31.254 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.247+0000 7f5ec1fa3640 1 Processor -- start 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.247+0000 7f5ec1fa3640 1 -- start start 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.247+0000 7f5ec1fa3640 1 librados: setting wanted keys 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.247+0000 7f5ec1fa3640 1 librados: calling monclient init 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.247+0000 7f5ec1fa3640 10 monclient: init 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.248+0000 7f5ec1fa3640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.248+0000 7f5ec1fa3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.248+0000 7f5ec1fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f5ebc0af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.248+0000 7f5ec1fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f5ebc0b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.248+0000 7f5ebaffd640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5ebc0b0860 0x7f5ebc1dc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:52354/0 (socket says 172.21.15.155:52354) 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.251+0000 7f5ec1fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f5ebc13ef30 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.251+0000 7f5ebaffd640 1 -- 172.21.15.155:0/3908972144 learned_addr learned my addr 172.21.15.155:0/3908972144 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.251+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc136190 con 0x7f5ebc0b0860 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.251+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc136470 con 0x7f5ebc13ef30 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.251+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ebc136750 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.255 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.251+0000 7f5ec1fa3640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.251+0000 7f5ec1fa3640 10 monclient(hunting): authenticate will time out at 1334.544835s 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 client.?.objecter ms_handle_connect 0x7f5ebc0b0860 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 client.?.objecter ms_handle_connect 0x7f5ebc13ef30 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 client.?.objecter ms_handle_connect 0x7f5ebc0af8e0 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613168859 0 0) 0x7f5ebc136190 con 0x7f5ebc0b0860 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea8004eb0 con 0x7f5ebc0b0860 2024-03-29T11:00:31.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571817260 0 0) 0x7f5ebc136750 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc136190 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392948925 0 0) 0x7f5ebc136470 con 0x7f5ebc13ef30 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.252+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ebc136750 con 0x7f5ebc13ef30 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2459373655 0 0) 0x7f5ebc136750 con 0x7f5ebc13ef30 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ebc136470 con 0x7f5ebc13ef30 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5eac001860 con 0x7f5ebc13ef30 2024-03-29T11:00:31.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): dump: 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:31.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3809351248 0 0) 0x7f5ebc136190 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ebc136750 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3431011739 0 0) 0x7f5ea8004eb0 con 0x7f5ebc0b0860 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ebc136190 con 0x7f5ebc0b0860 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5eb4001dd0 con 0x7f5ebc0af8e0 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): dump: 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:31.259 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5eb0001750 con 0x7f5ebc0b0860 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient(hunting): dump: 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:31.260 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3960448056 0 0) 0x7f5ebc136470 con 0x7f5ebc13ef30 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 >> v1:172.21.15.155:6790/0 conn(0x7f5ebc0af8e0 legacy=0x7f5ebc0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 >> v1:172.21.15.155:6789/0 conn(0x7f5ebc0b0860 legacy=0x7f5ebc1dc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 monclient: found mon.b 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0b4230 con 0x7f5ebc13ef30 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ec1fa3640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:00:31.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ec1fa3640 10 monclient: _renew_subs 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ec1fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.253+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc0b4420 con 0x7f5ebc13ef30 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ec1fa3640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ec1fa3640 10 monclient: _renew_subs 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ec1fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc1df170 con 0x7f5ebc13ef30 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ec1fa3640 1 librados: init done 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eac001c70 con 0x7f5ebc13ef30 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ea3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f5eac003790 con 0x7f5ebc13ef30 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ea3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.254+0000 7f5ea3fff640 10 monclient: dump: 2024-03-29T11:00:31.262 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.256+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f5eac019760 con 0x7f5ebc13ef30 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.256+0000 7f5eb9ffb640 4 set_mon_vals no callback set 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ec1fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f5eac083750 con 0x7f5ebc13ef30 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ea3fff640 10 client.4495.objecter ms_dispatch 0x7f5ebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:31.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ea3fff640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:31.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ea3fff640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:31.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ea3fff640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:00:31.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ec1fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:31.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ec1fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:31.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c005080 con 0x7f5ebc13ef30 2024-03-29T11:00:31.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.257+0000 7f5ebb7fe640 10 monclient: get_auth_request con 0x7f5ea806e0c0 auth_method 0 2024-03-29T11:00:31.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.259+0000 7f5ea3fff640 10 client.4495.objecter ms_handle_connect 0x7f5ea806e0c0 2024-03-29T11:00:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.266+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5eac04f030 con 0x7f5ebc13ef30 2024-03-29T11:00:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.266+0000 7f5ea3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:31.265 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.266+0000 7f5ea3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:31.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.591+0000 7f5ec1fa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:31.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.591+0000 7f5ec1fa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:31.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.591+0000 7f5ec1fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:31.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.591+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e9c005330 con 0x7f5ebc13ef30 2024-03-29T11:00:31.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.592+0000 7f5ea3fff640 1 -- 172.21.15.155:0/3908972144 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f5eac050f60 con 0x7f5ebc13ef30 2024-03-29T11:00:31.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.592+0000 7f5ea3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:31.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.592+0000 7f5ea3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:31.573 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:31.574 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:31.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.597+0000 7f5ec1fa3640 10 librados: watch_flush enter 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 10 librados: watch_flush exit 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 >> v1:172.21.15.183:6816/1412921548 conn(0x7f5ea806e0c0 legacy=0x7f5ea8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 10 monclient: shutdown 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 >> v1:172.21.15.183:6789/0 conn(0x7f5ebc13ef30 legacy=0x7f5ebc1e72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 shutdown_connections 2024-03-29T11:00:31.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.600+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 >> 172.21.15.155:0/3908972144 conn(0x7f5ebc06b660 msgr2=0x7f5ebc0ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:31.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.601+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 shutdown_connections 2024-03-29T11:00:31.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.601+0000 7f5ec1fa3640 1 -- 172.21.15.155:0/3908972144 wait complete. 2024-03-29T11:00:31.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:31.601+0000 7f5ec1fa3640 1 librados: shutdown 2024-03-29T11:00:31.598 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:38.405 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.574+0000 7f766ba2a640 10 monclient: get_monmap_and_config 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.574+0000 7f766ba2a640 10 monclient: build_initial_monmap 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.574+0000 7f766ba2a640 10 monclient: monmap: 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:38.575811+0000 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:38.575811+0000 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:38.555 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:38.556 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:38.556 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:38.556 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.575+0000 7f766ba2a640 1 Processor -- start 2024-03-29T11:00:38.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.575+0000 7f766ba2a640 1 -- start start 2024-03-29T11:00:38.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.575+0000 7f766ba2a640 10 monclient: init 2024-03-29T11:00:38.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:38.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:38.557 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 10 monclient(hunting): picked mon.noname-a con 0x7f76641422b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 10 monclient(hunting): picked mon.noname-c con 0x7f7664142690 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 10 monclient(hunting): picked mon.noname-b con 0x7f76640b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664142c80 con 0x7f76641422b0 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640aae40 con 0x7f7664142690 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766413fbd0 con 0x7f76640b17d0 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f766ba2a640 10 monclient(hunting): authenticate will time out at 1341.869218s 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f7668fc5640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7664142690 0x7f76640b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:41078/0 (socket says 172.21.15.155:41078) 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f76697c6640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f76641422b0 0x7f766413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:48830/0 (socket says 172.21.15.155:48830) 2024-03-29T11:00:38.559 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.576+0000 7f76697c6640 1 -- 172.21.15.155:0/2164827284 learned_addr learned my addr 172.21.15.155:0/2164827284 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968164576 0 0) 0x7f766413fbd0 con 0x7f76640b17d0 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7650003520 con 0x7f76640b17d0 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897690425 0 0) 0x7f7664142c80 con 0x7f76641422b0 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76500048a0 con 0x7f76641422b0 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3040571659 0 0) 0x7f7650003520 con 0x7f76640b17d0 2024-03-29T11:00:38.560 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 >> v1:172.21.15.183:6789/0 conn(0x7f7664142690 legacy=0x7f76640b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 >> v1:172.21.15.155:6789/0 conn(0x7f76641422b0 legacy=0x7f766413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 1 monclient: found mon.noname-b 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.577+0000 7f765bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664151940 con 0x7f76640b17d0 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 10 monclient: _finish_auth 0 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7654001670 con 0x7f76640b17d0 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 10 monclient: dump: 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:38.561 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f766ba2a640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f766ba2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7654001850 con 0x7f76640b17d0 2024-03-29T11:00:38.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f765bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:38.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f766ba2a640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:38.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.578+0000 7f766ba2a640 4 set_mon_vals no callback set 2024-03-29T11:00:38.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f765bfff640 1 -- 172.21.15.155:0/2164827284 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7654001670 con 0x7f76640b17d0 2024-03-29T11:00:38.570 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f765bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f765bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f765bfff640 10 monclient: dump: 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f766ba2a640 10 monclient: shutdown 2024-03-29T11:00:38.571 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f766ba2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f766ba2a640 1 -- 172.21.15.155:0/2164827284 >> v1:172.21.15.155:6790/0 conn(0x7f76640b17d0 legacy=0x7f76640b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f766ba2a640 1 -- 172.21.15.155:0/2164827284 shutdown_connections 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f766ba2a640 1 -- 172.21.15.155:0/2164827284 >> 172.21.15.155:0/2164827284 conn(0x7f7664066a00 msgr2=0x7f7664066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.579+0000 7f766ba2a640 1 -- 172.21.15.155:0/2164827284 shutdown_connections 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.580+0000 7f766ba2a640 1 -- 172.21.15.155:0/2164827284 wait complete. 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.580+0000 7f766ba2a640 10 monclient: build_initial_monmap 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.580+0000 7f766ba2a640 10 monclient: monmap: 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:38.572 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.581+0000 7f766ba2a640 1 librados: starting msgr at 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.581+0000 7f766ba2a640 1 librados: starting objecter 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.581+0000 7f766ba2a640 1 Processor -- start 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.581+0000 7f766ba2a640 1 -- start start 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.581+0000 7f766ba2a640 1 librados: setting wanted keys 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.581+0000 7f766ba2a640 1 librados: calling monclient init 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.581+0000 7f766ba2a640 10 monclient: init 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 10 monclient(hunting): picked mon.noname-c con 0x7f76640b17d0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 10 monclient(hunting): picked mon.noname-b con 0x7f76641422b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 10 monclient(hunting): picked mon.noname-a con 0x7f7664142690 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:38.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766406eea0 con 0x7f7664142690 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664131b90 con 0x7f76641422b0 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664132020 con 0x7f76640b17d0 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f766ba2a640 10 monclient(hunting): authenticate will time out at 1341.875380s 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f7668fc5640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f76641422b0 0x7f766412ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:41310/0 (socket says 172.21.15.155:41310) 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.582+0000 7f7668fc5640 1 -- 172.21.15.155:0/3673764430 learned_addr learned my addr 172.21.15.155:0/3673764430 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 client.?.objecter ms_handle_connect 0x7f76641422b0 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 client.?.objecter ms_handle_connect 0x7f7664142690 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 client.?.objecter ms_handle_connect 0x7f76640b17d0 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:38.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749016629 0 0) 0x7f7664131b90 con 0x7f76641422b0 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7648004eb0 con 0x7f76641422b0 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260784462 0 0) 0x7f766406eea0 con 0x7f7664142690 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.583+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7664131b90 con 0x7f7664142690 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2078774756 0 0) 0x7f7648004eb0 con 0x7f76641422b0 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f766406eea0 con 0x7f76641422b0 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f76600018d0 con 0x7f76641422b0 2024-03-29T11:00:38.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:38.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390520775 0 0) 0x7f7664132020 con 0x7f76640b17d0 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7648004eb0 con 0x7f76640b17d0 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4221732006 0 0) 0x7f7664131b90 con 0x7f7664142690 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7664132020 con 0x7f7664142690 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7654001630 con 0x7f7664142690 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.577 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.584+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863239234 0 0) 0x7f766406eea0 con 0x7f76641422b0 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 >> v1:172.21.15.155:6790/0 conn(0x7f76640b17d0 legacy=0x7f76641decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 >> v1:172.21.15.155:6789/0 conn(0x7f7664142690 legacy=0x7f76641e7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 1 monclient: found mon.b 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:38.578 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766406f0d0 con 0x7f76641422b0 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f7659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 10 monclient: _renew_subs 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664071f70 con 0x7f76641422b0 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 10 monclient: _renew_subs 2024-03-29T11:00:38.579 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7664132880 con 0x7f76641422b0 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.585+0000 7f766ba2a640 1 librados: init done 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.586+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76600047e0 con 0x7f76641422b0 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.586+0000 7f7659ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.586+0000 7f765b7fe640 4 set_mon_vals no callback set 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.586+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7660004af0 con 0x7f76641422b0 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.586+0000 7f7659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.586+0000 7f7659ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.586+0000 7f7659ffb640 10 monclient: dump: 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:38.580 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.587+0000 7f766ba2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.587+0000 7f766ba2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.587+0000 7f766ba2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.587+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7644005080 con 0x7f76641422b0 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.588+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f766001aac0 con 0x7f76641422b0 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.589+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f7660084ad0 con 0x7f76641422b0 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.589+0000 7f7659ffb640 10 client.4498.objecter ms_dispatch 0x7f7664000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.589+0000 7f7659ffb640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.589+0000 7f7659ffb640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.589+0000 7f7659ffb640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T11:00:38.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.589+0000 7f76697c6640 10 monclient: get_auth_request con 0x7f764806df70 auth_method 0 2024-03-29T11:00:38.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.590+0000 7f7659ffb640 10 client.4498.objecter ms_handle_connect 0x7f764806df70 2024-03-29T11:00:38.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.597+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76600503b0 con 0x7f76641422b0 2024-03-29T11:00:38.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.597+0000 7f7659ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:38.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.597+0000 7f7659ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:38.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.967+0000 7f766ba2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:38.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.967+0000 7f766ba2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:38.948 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.967+0000 7f766ba2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:00:38.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.967+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7644005330 con 0x7f76641422b0 2024-03-29T11:00:38.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.969+0000 7f7659ffb640 1 -- 172.21.15.155:0/3673764430 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f76600503b0 con 0x7f76641422b0 2024-03-29T11:00:38.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.969+0000 7f7659ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:38.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.969+0000 7f7659ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:38.950 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:38.950 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.972+0000 7f766ba2a640 10 librados: watch_flush enter 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.972+0000 7f766ba2a640 10 librados: watch_flush exit 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.972+0000 7f766ba2a640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.972+0000 7f766ba2a640 10 client.4498.objecter successfully canceled tick 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 >> v1:172.21.15.183:6816/1412921548 conn(0x7f764806df70 legacy=0x7f7648070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 10 monclient: shutdown 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:38.953 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 >> v1:172.21.15.183:6789/0 conn(0x7f76641422b0 legacy=0x7f766412ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:38.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 shutdown_connections 2024-03-29T11:00:38.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 >> 172.21.15.155:0/3673764430 conn(0x7f7664066a00 msgr2=0x7f76640b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:38.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 shutdown_connections 2024-03-29T11:00:38.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 1 -- 172.21.15.155:0/3673764430 wait complete. 2024-03-29T11:00:38.954 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:38.973+0000 7f766ba2a640 1 librados: shutdown 2024-03-29T11:00:38.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:45.776 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.926+0000 7fb6f89e6640 10 monclient: get_monmap_and_config 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.926+0000 7fb6f89e6640 10 monclient: build_initial_monmap 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.926+0000 7fb6f89e6640 10 monclient: monmap: 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:45.928092+0000 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:45.928092+0000 2024-03-29T11:00:45.907 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.927+0000 7fb6f89e6640 1 Processor -- start 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.927+0000 7fb6f89e6640 1 -- start start 2024-03-29T11:00:45.908 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.927+0000 7fb6f89e6640 10 monclient: init 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.928+0000 7fb6f89e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.928+0000 7fb6f89e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.928+0000 7fb6f89e6640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f41422b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.928+0000 7fb6f89e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f4142690 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f89e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f40b17d0 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f89e6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f4142c80 con 0x7fb6f4142690 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f2d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb6f40b17d0 0x7fb6f40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:38924/0 (socket says 172.21.15.155:38924) 2024-03-29T11:00:45.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f2d9d640 1 -- 172.21.15.155:0/684594150 learned_addr learned my addr 172.21.15.155:0/684594150 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f89e6640 1 -- 172.21.15.155:0/684594150 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f40aae40 con 0x7fb6f40b17d0 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f89e6640 1 -- 172.21.15.155:0/684594150 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f413fbd0 con 0x7fb6f41422b0 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f89e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f89e6640 10 monclient(hunting): authenticate will time out at 1349.222304s 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373801841 0 0) 0x7fb6f4142c80 con 0x7fb6f4142690 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc003520 con 0x7fb6f4142690 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727458048 0 0) 0x7fb6dc003520 con 0x7fb6f4142690 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.929+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 >> v1:172.21.15.155:6790/0 conn(0x7fb6f41422b0 legacy=0x7fb6f413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:45.914 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 >> v1:172.21.15.183:6789/0 conn(0x7fb6f40b17d0 legacy=0x7fb6f40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 10 monclient: _finish_hunting 0 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 1 monclient: found mon.noname-a 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f4150930 con 0x7fb6f4142690 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 10 monclient: _finish_auth 0 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 10 monclient: _check_auth_tickets 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f89e6640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb6ec001a10 con 0x7fb6f4142690 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f89e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:45.915 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f159a640 10 monclient: dump: 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.930+0000 7fb6f89e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6ec001c70 con 0x7fb6f4142690 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f159a640 1 -- 172.21.15.155:0/684594150 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb6ec002040 con 0x7fb6f4142690 2024-03-29T11:00:45.916 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f159a640 10 monclient: dump: 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f89e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.931+0000 7fb6f89e6640 4 set_mon_vals no callback set 2024-03-29T11:00:45.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.932+0000 7fb6f89e6640 10 monclient: shutdown 2024-03-29T11:00:45.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.932+0000 7fb6f89e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:45.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.932+0000 7fb6f89e6640 1 -- 172.21.15.155:0/684594150 >> v1:172.21.15.155:6789/0 conn(0x7fb6f4142690 legacy=0x7fb6f40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:45.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.932+0000 7fb6f89e6640 1 -- 172.21.15.155:0/684594150 shutdown_connections 2024-03-29T11:00:45.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.932+0000 7fb6f89e6640 1 -- 172.21.15.155:0/684594150 >> 172.21.15.155:0/684594150 conn(0x7fb6f4066a00 msgr2=0x7fb6f4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:45.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.932+0000 7fb6f89e6640 1 -- 172.21.15.155:0/684594150 shutdown_connections 2024-03-29T11:00:45.918 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.932+0000 7fb6f89e6640 1 -- 172.21.15.155:0/684594150 wait complete. 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.933+0000 7fb6f89e6640 10 monclient: build_initial_monmap 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.933+0000 7fb6f89e6640 10 monclient: monmap: 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:45.924 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.933+0000 7fb6f89e6640 1 librados: starting msgr at 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.933+0000 7fb6f89e6640 1 librados: starting objecter 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.934+0000 7fb6f89e6640 1 Processor -- start 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.934+0000 7fb6f89e6640 1 -- start start 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.934+0000 7fb6f89e6640 1 librados: setting wanted keys 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.934+0000 7fb6f89e6640 1 librados: calling monclient init 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.934+0000 7fb6f89e6640 10 monclient: init 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.934+0000 7fb6f89e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.934+0000 7fb6f89e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f40b17d0 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f41422b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f412e390 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f259c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb6f40b17d0 0x7fb6f412dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:42656/0 (socket says 172.21.15.155:42656) 2024-03-29T11:00:45.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f4141de0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f259c640 1 -- 172.21.15.155:0/2101307838 learned_addr learned my addr 172.21.15.155:0/2101307838 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f4141300 con 0x7fb6f412e390 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 client.?.objecter ms_handle_connect 0x7fb6f40b17d0 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f41305c0 con 0x7fb6f41422b0 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6f89e6640 10 monclient(hunting): authenticate will time out at 1349.228339s 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 client.?.objecter ms_handle_connect 0x7fb6f412e390 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646613849 0 0) 0x7fb6f4141de0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:45.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.935+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d4004eb0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2506208767 0 0) 0x7fb6d4004eb0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6f4141de0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb6e80017d0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:45.927 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2781120861 0 0) 0x7fb6f4141de0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.936+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 >> v1:172.21.15.155:6790/0 conn(0x7fb6f41422b0 legacy=0x7fb6f41db910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 >> v1:172.21.15.183:6789/0 conn(0x7fb6f412e390 legacy=0x7fb6f41e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 1 monclient: found mon.a 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f4140f90 con 0x7fb6f40b17d0 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:45.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6f89e6640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6f89e6640 10 monclient: _renew_subs 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6f89e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f406de90 con 0x7fb6f40b17d0 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6e8001d60 con 0x7fb6f40b17d0 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb6e8004a00 con 0x7fb6f40b17d0 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6f0d99640 4 set_mon_vals no callback set 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.939+0000 7fb6e2ffd640 10 monclient: dump: 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:45.929 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.940+0000 7fb6f89e6640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.940+0000 7fb6f89e6640 10 monclient: _renew_subs 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.940+0000 7fb6f89e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.940+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f406e1e0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.940+0000 7fb6f89e6640 1 librados: init done 2024-03-29T11:00:45.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.941+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fb6e801aa50 con 0x7fb6f40b17d0 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.942+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fb6e80548e0 con 0x7fb6f40b17d0 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.942+0000 7fb6e2ffd640 10 client.4524.objecter ms_dispatch 0x7fb6f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.942+0000 7fb6e2ffd640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.942+0000 7fb6e2ffd640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.942+0000 7fb6e2ffd640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.942+0000 7fb6f89e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.942+0000 7fb6f1d9b640 10 monclient: get_auth_request con 0x7fb6d406db30 auth_method 0 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.943+0000 7fb6f89e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.943+0000 7fb6f89e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:00:45.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.943+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f413ee10 con 0x7fb6f40b17d0 2024-03-29T11:00:45.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.946+0000 7fb6e2ffd640 10 client.4524.objecter ms_handle_connect 0x7fb6d406db30 2024-03-29T11:00:45.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.950+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6e8050220 con 0x7fb6f40b17d0 2024-03-29T11:00:45.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.950+0000 7fb6e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:45.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:45.950+0000 7fb6e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:46.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.263+0000 7fb6f89e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:46.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.263+0000 7fb6f89e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:46.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.263+0000 7fb6f89e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:00:46.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.263+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6f4141490 con 0x7fb6f40b17d0 2024-03-29T11:00:46.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.263+0000 7fb6e2ffd640 1 -- 172.21.15.155:0/2101307838 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fb6e8052320 con 0x7fb6f40b17d0 2024-03-29T11:00:46.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.263+0000 7fb6e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:46.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.263+0000 7fb6e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:46.245 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:46.245 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 10 librados: watch_flush enter 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 10 librados: watch_flush exit 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 10 client.4524.objecter successfully canceled tick 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 >> v1:172.21.15.183:6816/1412921548 conn(0x7fb6d406db30 legacy=0x7fb6d406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 10 monclient: shutdown 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 >> v1:172.21.15.155:6789/0 conn(0x7fb6f40b17d0 legacy=0x7fb6f412dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 shutdown_connections 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 >> 172.21.15.155:0/2101307838 conn(0x7fb6f4066a00 msgr2=0x7fb6f40b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 shutdown_connections 2024-03-29T11:00:46.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 1 -- 172.21.15.155:0/2101307838 wait complete. 2024-03-29T11:00:46.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:46.267+0000 7fb6f89e6640 1 librados: shutdown 2024-03-29T11:00:46.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:53.067 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:53.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.239+0000 7fd71a28f640 10 monclient: get_monmap_and_config 2024-03-29T11:00:53.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.239+0000 7fd71a28f640 10 monclient: build_initial_monmap 2024-03-29T11:00:53.219 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.239+0000 7fd71a28f640 10 monclient: monmap: 2024-03-29T11:00:53.219 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:00:53.240257+0000 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:00:53.240257+0000 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:00:53.220 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:53.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.240+0000 7fd71a28f640 1 Processor -- start 2024-03-29T11:00:53.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.240+0000 7fd71a28f640 1 -- start start 2024-03-29T11:00:53.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient: init 2024-03-29T11:00:53.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:53.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:53.221 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient(hunting): picked mon.noname-b con 0x7fd70c006e30 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient(hunting): picked mon.noname-a con 0x7fd70c035e00 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient(hunting): picked mon.noname-c con 0x7fd70c0c9670 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c035970 con 0x7fd70c035e00 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c036280 con 0x7fd70c0c9670 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c035640 con 0x7fd70c006e30 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.241+0000 7fd71a28f640 10 monclient(hunting): authenticate will time out at 1356.533774s 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.242+0000 7fd7137fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd70c035e00 0x7fd70c0c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50004/0 (socket says 172.21.15.155:50004) 2024-03-29T11:00:53.222 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.242+0000 7fd7137fe640 1 -- 172.21.15.155:0/2677121785 learned_addr learned my addr 172.21.15.155:0/2677121785 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:53.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.243+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306790837 0 0) 0x7fd70c035970 con 0x7fd70c035e00 2024-03-29T11:00:53.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.243+0000 7fd712ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:53.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.243+0000 7fd712ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:53.223 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.243+0000 7fd712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:53.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.243+0000 7fd712ffd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:00:53.224 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.243+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd704003520 con 0x7fd70c035e00 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.244+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2801768441 0 0) 0x7fd704003520 con 0x7fd70c035e00 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.244+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 >> v1:172.21.15.155:6790/0 conn(0x7fd70c006e30 legacy=0x7fd70c0cbcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.244+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 >> v1:172.21.15.183:6789/0 conn(0x7fd70c0c9670 legacy=0x7fd70c0d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.244+0000 7fd712ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 1 monclient: found mon.noname-a 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c034e20 con 0x7fd70c035e00 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:53.225 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd71404f730 con 0x7fd70c035e00 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: dump: 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:53.226 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd71a28f640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd71a28f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd714075f10 con 0x7fd70c035e00 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd71a28f640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd71a28f640 4 set_mon_vals no callback set 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 1 -- 172.21.15.155:0/2677121785 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd7140758b0 con 0x7fd70c035e00 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.245+0000 7fd712ffd640 10 monclient: dump: 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:53.227 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.228 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.228 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:53.228 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:53.228 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:53.228 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:53.228 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:53.228 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:53.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.246+0000 7fd71a28f640 10 monclient: shutdown 2024-03-29T11:00:53.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.246+0000 7fd71a28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:53.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.246+0000 7fd71a28f640 1 -- 172.21.15.155:0/2677121785 >> v1:172.21.15.155:6789/0 conn(0x7fd70c035e00 legacy=0x7fd70c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:53.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.247+0000 7fd71a28f640 1 -- 172.21.15.155:0/2677121785 shutdown_connections 2024-03-29T11:00:53.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.247+0000 7fd71a28f640 1 -- 172.21.15.155:0/2677121785 >> 172.21.15.155:0/2677121785 conn(0x7fd70c005c00 msgr2=0x7fd70c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:53.229 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.248+0000 7fd71a28f640 1 -- 172.21.15.155:0/2677121785 shutdown_connections 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.248+0000 7fd71a28f640 1 -- 172.21.15.155:0/2677121785 wait complete. 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.249+0000 7fd71a28f640 10 monclient: build_initial_monmap 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.249+0000 7fd71a28f640 10 monclient: monmap: 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:00:53.230 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:53.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.250+0000 7fd71a28f640 1 librados: starting msgr at 2024-03-29T11:00:53.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.250+0000 7fd71a28f640 1 librados: starting objecter 2024-03-29T11:00:53.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.250+0000 7fd71a28f640 1 Processor -- start 2024-03-29T11:00:53.231 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.250+0000 7fd71a28f640 1 -- start start 2024-03-29T11:00:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.250+0000 7fd71a28f640 1 librados: setting wanted keys 2024-03-29T11:00:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.250+0000 7fd71a28f640 1 librados: calling monclient init 2024-03-29T11:00:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.250+0000 7fd71a28f640 10 monclient: init 2024-03-29T11:00:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.251+0000 7fd71a28f640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:53.232 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.251+0000 7fd71a28f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.252+0000 7fd71a28f640 10 monclient(hunting): picked mon.noname-b con 0x7fd70c006e30 addr v1:172.21.15.183:6789/0 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.252+0000 7fd71a28f640 10 monclient(hunting): picked mon.noname-c con 0x7fd70c035e00 addr v1:172.21.15.155:6790/0 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.252+0000 7fd71a28f640 10 monclient(hunting): picked mon.noname-a con 0x7fd70c0c9670 addr v1:172.21.15.155:6789/0 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.252+0000 7fd71a28f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c033920 con 0x7fd70c0c9670 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.253+0000 7fd71a28f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c02e520 con 0x7fd70c006e30 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.253+0000 7fd71a28f640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70c033db0 con 0x7fd70c035e00 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.253+0000 7fd71a28f640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.253+0000 7fd71a28f640 10 monclient(hunting): authenticate will time out at 1356.544906s 2024-03-29T11:00:53.233 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.254+0000 7fd71882c640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd70c0c9670 0x7fd70c16cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:50504/0 (socket says 172.21.15.155:50504) 2024-03-29T11:00:53.234 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.254+0000 7fd71882c640 1 -- 172.21.15.155:0/760669318 learned_addr learned my addr 172.21.15.155:0/760669318 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:00:53.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.255+0000 7fd710ff9640 10 client.?.objecter ms_handle_connect 0x7fd70c0c9670 2024-03-29T11:00:53.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.255+0000 7fd710ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:53.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.255+0000 7fd710ff9640 10 client.?.objecter ms_handle_connect 0x7fd70c035e00 2024-03-29T11:00:53.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.255+0000 7fd710ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:53.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083523178 0 0) 0x7fd70c033920 con 0x7fd70c0c9670 2024-03-29T11:00:53.236 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f8004eb0 con 0x7fd70c0c9670 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129041728 0 0) 0x7fd70c033db0 con 0x7fd70c035e00 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70c033920 con 0x7fd70c035e00 2024-03-29T11:00:53.237 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316773391 0 0) 0x7fd70c033920 con 0x7fd70c035e00 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.256+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd70c033db0 con 0x7fd70c035e00 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd71404f400 con 0x7fd70c035e00 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:53.240 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4288564287 0 0) 0x7fd70c033db0 con 0x7fd70c035e00 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 >> v1:172.21.15.183:6789/0 conn(0x7fd70c006e30 legacy=0x7fd70c1647c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 >> v1:172.21.15.155:6789/0 conn(0x7fd70c0c9670 legacy=0x7fd70c16cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 monclient: found mon.c 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c032de0 con 0x7fd70c035e00 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:53.241 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 10 monclient: _renew_subs 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c02d150 con 0x7fd70c035e00 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 10 monclient: _renew_subs 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70c1650b0 con 0x7fd70c035e00 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd71a28f640 1 librados: init done 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd71404f730 con 0x7fd70c035e00 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fd714067d90 con 0x7fd70c035e00 2024-03-29T11:00:53.242 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.257+0000 7fd710ff9640 10 monclient: dump: 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.258+0000 7fd711ffb640 4 set_mon_vals no callback set 2024-03-29T11:00:53.243 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.260+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fd714067ef0 con 0x7fd70c035e00 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.260+0000 7fd71a28f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.261+0000 7fd713fff640 10 monclient: get_auth_request con 0x7fd6f806db00 auth_method 0 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.261+0000 7fd71a28f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.261+0000 7fd71a28f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.261+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f4005080 con 0x7fd70c035e00 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.261+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fd7140ff4e0 con 0x7fd70c035e00 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.262+0000 7fd710ff9640 10 client.4589.objecter ms_dispatch 0x7fd714000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.262+0000 7fd710ff9640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.262+0000 7fd710ff9640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:53.245 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.262+0000 7fd710ff9640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:00:53.246 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.263+0000 7fd710ff9640 10 client.4589.objecter ms_handle_connect 0x7fd6f806db00 2024-03-29T11:00:53.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.273+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7140cadc0 con 0x7fd70c035e00 2024-03-29T11:00:53.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.273+0000 7fd710ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:53.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.273+0000 7fd710ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:53.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.601+0000 7fd71a28f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:53.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.601+0000 7fd71a28f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:53.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.601+0000 7fd71a28f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:00:53.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.601+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6f4005330 con 0x7fd70c035e00 2024-03-29T11:00:53.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.602+0000 7fd710ff9640 1 -- 172.21.15.155:0/760669318 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fd714068220 con 0x7fd70c035e00 2024-03-29T11:00:53.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.602+0000 7fd710ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:53.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.602+0000 7fd710ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:53.583 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:00:53.583 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 10 librados: watch_flush enter 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 10 librados: watch_flush exit 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 >> v1:172.21.15.183:6816/1412921548 conn(0x7fd6f806db00 legacy=0x7fd6f806ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 10 monclient: shutdown 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 >> v1:172.21.15.155:6790/0 conn(0x7fd70c035e00 legacy=0x7fd70c0c0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 shutdown_connections 2024-03-29T11:00:53.586 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.606+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 >> 172.21.15.155:0/760669318 conn(0x7fd70c005c00 msgr2=0x7fd70c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:53.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.607+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 shutdown_connections 2024-03-29T11:00:53.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.608+0000 7fd71a28f640 1 -- 172.21.15.155:0/760669318 wait complete. 2024-03-29T11:00:53.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:00:53.608+0000 7fd71a28f640 1 librados: shutdown 2024-03-29T11:00:53.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:00.413 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:00.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.582+0000 7f92642b3640 10 monclient: get_monmap_and_config 2024-03-29T11:01:00.562 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.582+0000 7f92642b3640 10 monclient: build_initial_monmap 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.582+0000 7f92642b3640 10 monclient: monmap: 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:00.583475+0000 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:00.583475+0000 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:00.563 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:00.565 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.585+0000 7f92642b3640 1 Processor -- start 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.586+0000 7f92642b3640 1 -- start start 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.586+0000 7f92642b3640 10 monclient: init 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 10 monclient(hunting): picked mon.noname-b con 0x7f925c0b0770 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 10 monclient(hunting): picked mon.noname-c con 0x7f925c14a700 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 10 monclient(hunting): picked mon.noname-a con 0x7f925c0ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:00.567 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c0aba70 con 0x7f925c0ac5a0 2024-03-29T11:01:00.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c0aecb0 con 0x7f925c14a700 2024-03-29T11:01:00.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c0af150 con 0x7f925c0b0770 2024-03-29T11:01:00.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:00.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.587+0000 7f92642b3640 10 monclient(hunting): authenticate will time out at 1363.879639s 2024-03-29T11:01:00.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926204f640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f925c0b0770 0x7f925c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:57806/0 (socket says 172.21.15.155:57806) 2024-03-29T11:01:00.568 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926204f640 1 -- 172.21.15.155:0/2546744569 learned_addr learned my addr 172.21.15.155:0/2546744569 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442032934 0 0) 0x7f925c0af150 con 0x7f925c0b0770 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 10 monclient(hunting): _negotiate 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9248003520 con 0x7f925c0b0770 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619470944 0 0) 0x7f9248003520 con 0x7f925c0b0770 2024-03-29T11:01:00.569 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.588+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 >> v1:172.21.15.183:6789/0 conn(0x7f925c14a700 legacy=0x7f925c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 >> v1:172.21.15.155:6789/0 conn(0x7f925c0ac5a0 legacy=0x7f925c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 10 monclient: _finish_hunting 0 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 1 monclient: found mon.noname-b 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c0ac9d0 con 0x7f925c0b0770 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 10 monclient: _finish_auth 0 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 10 monclient: _check_auth_tickets 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:00.573 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9258001690 con 0x7f925c0b0770 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f926104d640 10 monclient: dump: 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:00.574 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f92642b3640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.591+0000 7f92642b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.592+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9258001860 con 0x7f925c0b0770 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.592+0000 7f926104d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.592+0000 7f926104d640 1 -- 172.21.15.155:0/2546744569 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9258001cb0 con 0x7f925c0b0770 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.592+0000 7f926104d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.592+0000 7f926104d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.592+0000 7f926104d640 10 monclient: dump: 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:00.575 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.593+0000 7f92642b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.593+0000 7f92642b3640 4 set_mon_vals no callback set 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.593+0000 7f92642b3640 10 monclient: shutdown 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.593+0000 7f92642b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:00.576 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.594+0000 7f92642b3640 1 -- 172.21.15.155:0/2546744569 >> v1:172.21.15.155:6790/0 conn(0x7f925c0b0770 legacy=0x7f925c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.594+0000 7f92642b3640 1 -- 172.21.15.155:0/2546744569 shutdown_connections 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.594+0000 7f92642b3640 1 -- 172.21.15.155:0/2546744569 >> 172.21.15.155:0/2546744569 conn(0x7f925c06b660 msgr2=0x7f925c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.595+0000 7f92642b3640 1 -- 172.21.15.155:0/2546744569 shutdown_connections 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.595+0000 7f92642b3640 1 -- 172.21.15.155:0/2546744569 wait complete. 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.596+0000 7f92642b3640 10 monclient: build_initial_monmap 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.596+0000 7f92642b3640 10 monclient: monmap: 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:00.580 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 1 librados: starting msgr at 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 1 librados: starting objecter 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 1 Processor -- start 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 1 -- start start 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 1 librados: setting wanted keys 2024-03-29T11:01:00.581 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 1 librados: calling monclient init 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 10 monclient: init 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.597+0000 7f92642b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 10 monclient(hunting): picked mon.noname-a con 0x7f925c0ac5a0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 10 monclient(hunting): picked mon.noname-b con 0x7f925c0b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 10 monclient(hunting): picked mon.noname-c con 0x7f925c14a700 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c127e70 con 0x7f925c0ac5a0 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c125af0 con 0x7f925c0b0770 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c126f30 con 0x7f925c14a700 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f92642b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f9262850640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f925c14a700 0x7f925c1db5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:40910/0 (socket says 172.21.15.155:40910) 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f9262850640 1 -- 172.21.15.155:0/1765819685 learned_addr learned my addr 172.21.15.155:0/1765819685 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:00.582 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f9252ffd640 10 client.?.objecter ms_handle_connect 0x7f925c14a700 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f9252ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.598+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784261443 0 0) 0x7f925c126f30 con 0x7f925c14a700 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f92642b3640 10 monclient(hunting): authenticate will time out at 1363.890899s 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9240004eb0 con 0x7f925c14a700 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 619057450 0 0) 0x7f9240004eb0 con 0x7f925c14a700 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f925c126f30 con 0x7f925c14a700 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9254001c90 con 0x7f925c14a700 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:00.583 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): dump: 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3778145675 0 0) 0x7f925c126f30 con 0x7f925c14a700 2024-03-29T11:01:00.584 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.599+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 >> v1:172.21.15.183:6789/0 conn(0x7f925c0b0770 legacy=0x7f925c144bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:00.587 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 >> v1:172.21.15.155:6789/0 conn(0x7f925c0ac5a0 legacy=0x7f925c126540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 1 monclient: found mon.c 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c126c50 con 0x7f925c14a700 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 client.?.objecter ms_handle_connect 0x7f925c0ac5a0 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f92642b3640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9254002010 con 0x7f925c14a700 2024-03-29T11:01:00.588 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f92642b3640 10 monclient: _renew_subs 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f92642b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c145900 con 0x7f925c14a700 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9254003de0 con 0x7f925c14a700 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f92642b3640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9252ffd640 10 monclient: dump: 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:00.589 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.602+0000 7f9253fff640 4 set_mon_vals no callback set 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.603+0000 7f92642b3640 10 monclient: _renew_subs 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.603+0000 7f92642b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.603+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c1454e0 con 0x7f925c14a700 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.603+0000 7f92642b3640 1 librados: init done 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.604+0000 7f92642b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.604+0000 7f92642b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.604+0000 7f92642b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.604+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c0b0b50 con 0x7f925c14a700 2024-03-29T11:01:00.590 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.607+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f9254003f40 con 0x7f925c14a700 2024-03-29T11:01:00.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.609+0000 7f926204f640 10 monclient: get_auth_request con 0x7f924006db50 auth_method 0 2024-03-29T11:01:00.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.609+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f9254054d00 con 0x7f925c14a700 2024-03-29T11:01:00.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.609+0000 7f9252ffd640 10 client.4595.objecter ms_dispatch 0x7f925c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:00.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.609+0000 7f9252ffd640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:00.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.609+0000 7f9252ffd640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:00.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.609+0000 7f9252ffd640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:01:00.591 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.609+0000 7f9252ffd640 10 client.4595.objecter ms_handle_connect 0x7f924006db50 2024-03-29T11:01:00.594 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.615+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9254050810 con 0x7f925c14a700 2024-03-29T11:01:00.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.615+0000 7f9252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:00.595 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.615+0000 7f9252ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:00.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.941+0000 7f92642b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:00.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.941+0000 7f92642b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:00.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.941+0000 7f92642b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:00.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.941+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f925c145a90 con 0x7f925c14a700 2024-03-29T11:01:00.924 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:00.924 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:00.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.943+0000 7f9252ffd640 1 -- 172.21.15.155:0/1765819685 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f925c145a90 con 0x7f925c14a700 2024-03-29T11:01:00.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.943+0000 7f9252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:00.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.943+0000 7f9252ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:00.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.946+0000 7f92642b3640 10 librados: watch_flush enter 2024-03-29T11:01:00.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.947+0000 7f92642b3640 10 librados: watch_flush exit 2024-03-29T11:01:00.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.947+0000 7f92642b3640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:01:00.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.947+0000 7f92642b3640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:01:00.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 >> v1:172.21.15.183:6816/1412921548 conn(0x7f924006db50 legacy=0x7f9240070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:00.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 10 monclient: shutdown 2024-03-29T11:01:00.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:00.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 >> v1:172.21.15.155:6790/0 conn(0x7f925c14a700 legacy=0x7f925c1db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:00.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 shutdown_connections 2024-03-29T11:01:00.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 >> 172.21.15.155:0/1765819685 conn(0x7f925c06b660 msgr2=0x7f925c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:00.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 shutdown_connections 2024-03-29T11:01:00.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.950+0000 7f92642b3640 1 -- 172.21.15.155:0/1765819685 wait complete. 2024-03-29T11:01:00.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:00.951+0000 7f92642b3640 1 librados: shutdown 2024-03-29T11:01:00.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:07.756 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:07.909 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.929+0000 7f4a6bd6b640 10 monclient: get_monmap_and_config 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.929+0000 7f4a6bd6b640 10 monclient: build_initial_monmap 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.929+0000 7f4a6bd6b640 10 monclient: monmap: 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:07.929927+0000 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:07.929927+0000 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:07.910 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:07.911 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:07.911 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.932+0000 7f4a6bd6b640 1 Processor -- start 2024-03-29T11:01:07.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.932+0000 7f4a6bd6b640 1 -- start start 2024-03-29T11:01:07.912 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.932+0000 7f4a6bd6b640 10 monclient: init 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 10 monclient(hunting): picked mon.noname-c con 0x7f4a640a9ac0 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6413ee90 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 10 monclient(hunting): picked mon.noname-a con 0x7f4a640af900 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a640afd10 con 0x7f4a640af900 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a640af6f0 con 0x7f4a640a9ac0 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a640aac80 con 0x7f4a6413ee90 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:07.913 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.933+0000 7f4a6bd6b640 10 monclient(hunting): authenticate will time out at 1371.225588s 2024-03-29T11:01:07.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.937+0000 7f4a69b07640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4a640a9ac0 0x7f4a640adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:52037/0 (socket says 172.21.15.155:52037) 2024-03-29T11:01:07.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.937+0000 7f4a69b07640 1 -- 172.21.15.155:0/2510829220 learned_addr learned my addr 172.21.15.155:0/2510829220 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:07.917 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.937+0000 7f4a69306640 1 --1- 172.21.15.155:0/2510829220 >> v1:172.21.15.155:6790/0 conn(0x7f4a6413ee90 0x7f4a6413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:56520/0 (socket says 172.21.15.155:56520) 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168914815 0 0) 0x7f4a640aac80 con 0x7f4a6413ee90 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): _negotiate 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a50003520 con 0x7f4a6413ee90 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373640453 0 0) 0x7f4a640af6f0 con 0x7f4a640a9ac0 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): _negotiate 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:01:07.920 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a640aac80 con 0x7f4a640a9ac0 2024-03-29T11:01:07.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 577437701 0 0) 0x7f4a640aac80 con 0x7f4a640a9ac0 2024-03-29T11:01:07.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 >> v1:172.21.15.155:6790/0 conn(0x7f4a6413ee90 legacy=0x7f4a6413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:07.921 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.939+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 >> v1:172.21.15.155:6789/0 conn(0x7f4a640af900 legacy=0x7f4a64149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:07.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 10 monclient: _finish_hunting 0 2024-03-29T11:01:07.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 1 monclient: found mon.noname-c 2024-03-29T11:01:07.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:07.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:01:07.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a640aaf60 con 0x7f4a640a9ac0 2024-03-29T11:01:07.923 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 10 monclient: _finish_auth 0 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 10 monclient: _check_auth_tickets 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.941+0000 7f4a68b05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a6bd6b640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a6bd6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4a60001670 con 0x7f4a640a9ac0 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 10 monclient: dump: 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:07.924 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a6bd6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a60001890 con 0x7f4a640a9ac0 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a6bd6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a6bd6b640 4 set_mon_vals no callback set 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 1 -- 172.21.15.155:0/2510829220 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4a60001c20 con 0x7f4a640a9ac0 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:07.925 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.942+0000 7f4a68b05640 10 monclient: dump: 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.943+0000 7f4a6bd6b640 10 monclient: shutdown 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.943+0000 7f4a6bd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.943+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/2510829220 >> v1:172.21.15.183:6789/0 conn(0x7f4a640a9ac0 legacy=0x7f4a640adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:07.926 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.943+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/2510829220 shutdown_connections 2024-03-29T11:01:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.943+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/2510829220 >> 172.21.15.155:0/2510829220 conn(0x7f4a640b35e0 msgr2=0x7f4a640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.944+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/2510829220 shutdown_connections 2024-03-29T11:01:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.944+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/2510829220 wait complete. 2024-03-29T11:01:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.945+0000 7f4a6bd6b640 10 monclient: build_initial_monmap 2024-03-29T11:01:07.927 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.945+0000 7f4a6bd6b640 10 monclient: monmap: 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.946+0000 7f4a6bd6b640 1 librados: starting msgr at 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.946+0000 7f4a6bd6b640 1 librados: starting objecter 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.946+0000 7f4a6bd6b640 1 Processor -- start 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.947+0000 7f4a6bd6b640 1 -- start start 2024-03-29T11:01:07.928 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.947+0000 7f4a6bd6b640 1 librados: setting wanted keys 2024-03-29T11:01:07.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.947+0000 7f4a6bd6b640 1 librados: calling monclient init 2024-03-29T11:01:07.929 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.947+0000 7f4a6bd6b640 10 monclient: init 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 10 monclient(hunting): picked mon.noname-a con 0x7f4a640a9ac0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 10 monclient(hunting): picked mon.noname-c con 0x7f4a640af900 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6413ee90 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a64135760 con 0x7f4a640a9ac0 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a641369f0 con 0x7f4a6413ee90 2024-03-29T11:01:07.930 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a641e1790 con 0x7f4a640af900 2024-03-29T11:01:07.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:07.931 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.948+0000 7f4a6bd6b640 10 monclient(hunting): authenticate will time out at 1371.240518s 2024-03-29T11:01:07.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.952+0000 7f4a69306640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f4a640af900 0x7f4a641381e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:58070/0 (socket says 172.21.15.155:58070) 2024-03-29T11:01:07.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.952+0000 7f4a69306640 1 -- 172.21.15.155:0/3147280559 learned_addr learned my addr 172.21.15.155:0/3147280559 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:07.932 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.953+0000 7f4a5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4a640af900 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.953+0000 7f4a5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.953+0000 7f4a5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4a6413ee90 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.953+0000 7f4a5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.953+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561022595 0 0) 0x7f4a641e1790 con 0x7f4a640af900 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:07.938 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a48004eb0 con 0x7f4a640af900 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366460755 0 0) 0x7f4a641369f0 con 0x7f4a6413ee90 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.954+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a641e1790 con 0x7f4a6413ee90 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.955+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230394924 0 0) 0x7f4a48004eb0 con 0x7f4a640af900 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.955+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a641369f0 con 0x7f4a640af900 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.955+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047584766 0 0) 0x7f4a641e1790 con 0x7f4a6413ee90 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.955+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a48004eb0 con 0x7f4a6413ee90 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.955+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4a54001970 con 0x7f4a640af900 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.955+0000 7f4a5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.955+0000 7f4a5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:07.939 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient(hunting): dump: 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4a5c001a60 con 0x7f4a6413ee90 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:07.940 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient(hunting): dump: 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1998935910 0 0) 0x7f4a641369f0 con 0x7f4a640af900 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 >> v1:172.21.15.183:6789/0 conn(0x7f4a6413ee90 legacy=0x7f4a641e1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 >> v1:172.21.15.155:6789/0 conn(0x7f4a640a9ac0 legacy=0x7f4a641361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:07.941 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 1 monclient: found mon.c 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a641e1920 con 0x7f4a640af900 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 10 monclient: _renew_subs 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a641e3dc0 con 0x7f4a640af900 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 10 monclient: _renew_subs 2024-03-29T11:01:07.942 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:07.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a641e42a0 con 0x7f4a640af900 2024-03-29T11:01:07.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.956+0000 7f4a6bd6b640 1 librados: init done 2024-03-29T11:01:07.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.958+0000 7f4a6bd6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:07.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.958+0000 7f4a6bd6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:07.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.958+0000 7f4a6bd6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:07.943 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.958+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6413bd00 con 0x7f4a640af900 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.963+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a540047e0 con 0x7f4a640af900 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.963+0000 7f4a5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.963+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f4a54004950 con 0x7f4a640af900 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.963+0000 7f4a5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.963+0000 7f4a5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.963+0000 7f4a5a7fc640 10 monclient: dump: 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:07.944 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:07.945 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:07.945 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:07.945 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:07.945 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:07.945 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:07.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.964+0000 7f4a5bfff640 4 set_mon_vals no callback set 2024-03-29T11:01:07.945 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.964+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f4a5401a920 con 0x7f4a640af900 2024-03-29T11:01:07.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.965+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f4a54084770 con 0x7f4a640af900 2024-03-29T11:01:07.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.965+0000 7f4a5a7fc640 10 client.4601.objecter ms_dispatch 0x7f4a64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:07.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.965+0000 7f4a5a7fc640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:07.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.965+0000 7f4a5a7fc640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:07.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.965+0000 7f4a5a7fc640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:01:07.946 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.966+0000 7f4a69b07640 10 monclient: get_auth_request con 0x7f4a4806e0a0 auth_method 0 2024-03-29T11:01:07.947 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.967+0000 7f4a5a7fc640 10 client.4601.objecter ms_handle_connect 0x7f4a4806e0a0 2024-03-29T11:01:07.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.969+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a5404fe80 con 0x7f4a640af900 2024-03-29T11:01:07.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.969+0000 7f4a5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:07.949 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:07.969+0000 7f4a5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:08.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.292+0000 7f4a6bd6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:08.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.292+0000 7f4a6bd6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:08.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.292+0000 7f4a6bd6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:08.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.292+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a6406dfd0 con 0x7f4a640af900 2024-03-29T11:01:08.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.293+0000 7f4a5a7fc640 1 -- 172.21.15.155:0/3147280559 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f4a54057ba0 con 0x7f4a640af900 2024-03-29T11:01:08.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.293+0000 7f4a5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:08.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.293+0000 7f4a5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:08.273 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:08.273 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:08.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.296+0000 7f4a6bd6b640 10 librados: watch_flush enter 2024-03-29T11:01:08.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.297+0000 7f4a6bd6b640 10 librados: watch_flush exit 2024-03-29T11:01:08.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.297+0000 7f4a6bd6b640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:01:08.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.297+0000 7f4a6bd6b640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.297+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 >> v1:172.21.15.183:6816/1412921548 conn(0x7f4a4806e0a0 legacy=0x7f4a48070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.297+0000 7f4a6bd6b640 10 monclient: shutdown 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.297+0000 7f4a6bd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.297+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 >> v1:172.21.15.155:6790/0 conn(0x7f4a640af900 legacy=0x7f4a641381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.298+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 shutdown_connections 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.298+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 >> 172.21.15.155:0/3147280559 conn(0x7f4a640b35e0 msgr2=0x7f4a6413b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.298+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 shutdown_connections 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.298+0000 7f4a6bd6b640 1 -- 172.21.15.155:0/3147280559 wait complete. 2024-03-29T11:01:08.278 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:08.298+0000 7f4a6bd6b640 1 librados: shutdown 2024-03-29T11:01:08.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:15.102 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.268+0000 7f2b1a793640 10 monclient: get_monmap_and_config 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.268+0000 7f2b1a793640 10 monclient: build_initial_monmap 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.268+0000 7f2b1a793640 10 monclient: monmap: 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:15.269104+0000 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:15.269104+0000 2024-03-29T11:01:15.248 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.269+0000 7f2b1a793640 1 Processor -- start 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.269+0000 7f2b1a793640 1 -- start start 2024-03-29T11:01:15.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.269+0000 7f2b1a793640 10 monclient: init 2024-03-29T11:01:15.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.270+0000 7f2b1a793640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:15.256 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.270+0000 7f2b1a793640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.270+0000 7f2b1a793640 10 monclient(hunting): picked mon.noname-c con 0x7f2b140b0860 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b1a793640 10 monclient(hunting): picked mon.noname-a con 0x7f2b1413ef30 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b1a793640 10 monclient(hunting): picked mon.noname-b con 0x7f2b140af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b13fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2b140b0860 0x7f2b1413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:46620/0 (socket says 172.21.15.155:46620) 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b13fff640 1 -- 172.21.15.155:0/2583970285 learned_addr learned my addr 172.21.15.155:0/2583970285 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140b0c40 con 0x7f2b1413ef30 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140aed10 con 0x7f2b140b0860 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140afd50 con 0x7f2b140af8e0 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b1a793640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.271+0000 7f2b1a793640 10 monclient(hunting): authenticate will time out at 1378.563729s 2024-03-29T11:01:15.257 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19351599 0 0) 0x7f2b140b0c40 con 0x7f2b1413ef30 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af8003520 con 0x7f2b1413ef30 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924198023 0 0) 0x7f2b140afd50 con 0x7f2b140af8e0 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b140b0c40 con 0x7f2b140af8e0 2024-03-29T11:01:15.258 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 865659999 0 0) 0x7f2af8003520 con 0x7f2b1413ef30 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 >> v1:172.21.15.155:6790/0 conn(0x7f2b140af8e0 legacy=0x7f2b14149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 >> v1:172.21.15.183:6789/0 conn(0x7f2b140b0860 legacy=0x7f2b1413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 monclient: found mon.noname-a 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140ab1b0 con 0x7f2b1413ef30 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2b00001690 con 0x7f2b1413ef30 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:15.259 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.272+0000 7f2b12ffd640 10 monclient: dump: 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b1a793640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b1a793640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:15.260 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b000018e0 con 0x7f2b1413ef30 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b12ffd640 1 -- 172.21.15.155:0/2583970285 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2b00001c70 con 0x7f2b1413ef30 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b12ffd640 10 monclient: dump: 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:15.261 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b1a793640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.273+0000 7f2b1a793640 4 set_mon_vals no callback set 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.274+0000 7f2b1a793640 10 monclient: shutdown 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.274+0000 7f2b1a793640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.274+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 >> v1:172.21.15.155:6789/0 conn(0x7f2b1413ef30 legacy=0x7f2b140af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.274+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 shutdown_connections 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.274+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 >> 172.21.15.155:0/2583970285 conn(0x7f2b1406b660 msgr2=0x7f2b1406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.274+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 shutdown_connections 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.275+0000 7f2b1a793640 1 -- 172.21.15.155:0/2583970285 wait complete. 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.276+0000 7f2b1a793640 10 monclient: build_initial_monmap 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.276+0000 7f2b1a793640 10 monclient: monmap: 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:15.262 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.276+0000 7f2b1a793640 1 librados: starting msgr at 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.276+0000 7f2b1a793640 1 librados: starting objecter 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.276+0000 7f2b1a793640 1 Processor -- start 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.277+0000 7f2b1a793640 1 -- start start 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.277+0000 7f2b1a793640 1 librados: setting wanted keys 2024-03-29T11:01:15.263 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.277+0000 7f2b1a793640 1 librados: calling monclient init 2024-03-29T11:01:15.264 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.277+0000 7f2b1a793640 10 monclient: init 2024-03-29T11:01:15.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:15.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:15.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 10 monclient(hunting): picked mon.noname-b con 0x7f2b140af8e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:15.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 10 monclient(hunting): picked mon.noname-a con 0x7f2b140b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:15.268 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 10 monclient(hunting): picked mon.noname-c con 0x7f2b1413ef30 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140b3d00 con 0x7f2b140b0860 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140b3f70 con 0x7f2b140af8e0 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140b1c20 con 0x7f2b1413ef30 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b1a793640 10 monclient(hunting): authenticate will time out at 1378.570333s 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b137fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2b140b0860 0x7f2b140b2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:35092/0 (socket says 172.21.15.155:35092) 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b137fe640 1 -- 172.21.15.155:0/1396147149 learned_addr learned my addr 172.21.15.155:0/1396147149 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 client.?.objecter ms_handle_connect 0x7f2b140b0860 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 client.?.objecter ms_handle_connect 0x7f2b1413ef30 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:15.269 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467330371 0 0) 0x7f2b140b3d00 con 0x7f2b140b0860 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.278+0000 7f2b10ff9640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc004eb0 con 0x7f2b140b0860 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734312235 0 0) 0x7f2b140b1c20 con 0x7f2b1413ef30 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b140b3d00 con 0x7f2b1413ef30 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026832362 0 0) 0x7f2afc004eb0 con 0x7f2b140b0860 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2afc0064f0 con 0x7f2b140b0860 2024-03-29T11:01:15.270 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2b00001cb0 con 0x7f2b140b0860 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:15.271 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 356766800 0 0) 0x7f2b140b3d00 con 0x7f2b1413ef30 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2afc004eb0 con 0x7f2b1413ef30 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2b04001870 con 0x7f2b1413ef30 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.279+0000 7f2b10ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:15.272 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 698234636 0 0) 0x7f2afc0064f0 con 0x7f2b140b0860 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 >> v1:172.21.15.155:6790/0 conn(0x7f2b1413ef30 legacy=0x7f2b141ddc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 >> v1:172.21.15.183:6789/0 conn(0x7f2b140af8e0 legacy=0x7f2b140b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 1 monclient: found mon.a 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14071a80 con 0x7f2b140b0860 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: _finish_auth 0 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 10 monclient: _renew_subs 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b141d73c0 con 0x7f2b140b0860 2024-03-29T11:01:15.273 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 10 monclient: _renew_subs 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b141d6b80 con 0x7f2b140b0860 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b1a793640 1 librados: init done 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b000038a0 con 0x7f2b140b0860 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f2b00003c30 con 0x7f2b140b0860 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b10ff9640 10 monclient: dump: 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:15.274 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.280+0000 7f2b127fc640 4 set_mon_vals no callback set 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.282+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f2b00003d90 con 0x7f2b140b0860 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.282+0000 7f2b1a793640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.282+0000 7f2b1a793640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.282+0000 7f2b1a793640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.282+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b14071c70 con 0x7f2b140b0860 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.283+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f2b00084ba0 con 0x7f2b140b0860 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.283+0000 7f2b10ff9640 10 client.4539.objecter ms_dispatch 0x7f2b14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:15.275 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.283+0000 7f2b10ff9640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:15.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.283+0000 7f2b10ff9640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:15.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.283+0000 7f2b10ff9640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:01:15.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.283+0000 7f2b13fff640 10 monclient: get_auth_request con 0x7f2afc06e1d0 auth_method 0 2024-03-29T11:01:15.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.284+0000 7f2b10ff9640 10 client.4539.objecter ms_handle_connect 0x7f2afc06e1d0 2024-03-29T11:01:15.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.292+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b14071c70 con 0x7f2b140b0860 2024-03-29T11:01:15.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.292+0000 7f2b10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:15.276 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.292+0000 7f2b10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:15.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.667+0000 7f2b1a793640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:15.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.667+0000 7f2b1a793640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:15.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.667+0000 7f2b1a793640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:15.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.667+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b14071c10 con 0x7f2b140b0860 2024-03-29T11:01:15.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.668+0000 7f2b10ff9640 1 -- 172.21.15.155:0/1396147149 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f2b00050480 con 0x7f2b140b0860 2024-03-29T11:01:15.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.668+0000 7f2b10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:15.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.668+0000 7f2b10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:15.648 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:15.648 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:15.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.671+0000 7f2b1a793640 10 librados: watch_flush enter 2024-03-29T11:01:15.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.671+0000 7f2b1a793640 10 librados: watch_flush exit 2024-03-29T11:01:15.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.671+0000 7f2b1a793640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:01:15.651 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.671+0000 7f2b1a793640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 >> v1:172.21.15.183:6816/1412921548 conn(0x7f2afc06e1d0 legacy=0x7f2afc070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 10 monclient: shutdown 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 >> v1:172.21.15.155:6789/0 conn(0x7f2b140b0860 legacy=0x7f2b140b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 shutdown_connections 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 >> 172.21.15.155:0/1396147149 conn(0x7f2b1406b660 msgr2=0x7f2b1413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 shutdown_connections 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 1 -- 172.21.15.155:0/1396147149 wait complete. 2024-03-29T11:01:15.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:15.672+0000 7f2b1a793640 1 librados: shutdown 2024-03-29T11:01:15.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:22.474 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:22.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.660+0000 7fecf2601640 10 monclient: get_monmap_and_config 2024-03-29T11:01:22.641 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.660+0000 7fecf2601640 10 monclient: build_initial_monmap 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.660+0000 7fecf2601640 10 monclient: monmap: 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:22.662574+0000 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:22.662574+0000 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:22.642 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:22.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.665+0000 7fecf2601640 1 Processor -- start 2024-03-29T11:01:22.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.665+0000 7fecf2601640 1 -- start start 2024-03-29T11:01:22.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.665+0000 7fecf2601640 10 monclient: init 2024-03-29T11:01:22.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:22.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:22.647 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 10 monclient(hunting): picked mon.noname-a con 0x7fecec0b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 10 monclient(hunting): picked mon.noname-c con 0x7fecec13ef30 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 10 monclient(hunting): picked mon.noname-b con 0x7fecec0af8e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecec0b0c40 con 0x7fecec0b0860 2024-03-29T11:01:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecec0aed10 con 0x7fecec13ef30 2024-03-29T11:01:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecec0afd50 con 0x7fecec0af8e0 2024-03-29T11:01:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:22.648 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.666+0000 7fecf2601640 10 monclient(hunting): authenticate will time out at 1385.959266s 2024-03-29T11:01:22.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.671+0000 7fecebfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fecec0b0860 0x7fecec13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:51605/0 (socket says 172.21.15.155:51605) 2024-03-29T11:01:22.652 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.671+0000 7fecebfff640 1 -- 172.21.15.155:0/3112242728 learned_addr learned my addr 172.21.15.155:0/3112242728 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:22.653 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004395424 0 0) 0x7fecec0b0c40 con 0x7fecec0b0860 2024-03-29T11:01:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:01:22.654 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd8003520 con 0x7fecec0b0860 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391379009 0 0) 0x7fecec0aed10 con 0x7fecec13ef30 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.673+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecec0b0c40 con 0x7fecec13ef30 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2685194187 0 0) 0x7fecec0b0c40 con 0x7fecec13ef30 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 >> v1:172.21.15.155:6790/0 conn(0x7fecec0af8e0 legacy=0x7fecec149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 >> v1:172.21.15.155:6789/0 conn(0x7fecec0b0860 legacy=0x7fecec13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 monclient: found mon.noname-c 2024-03-29T11:01:22.655 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecec0ab1b0 con 0x7fecec13ef30 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fecdc001670 con 0x7fecec13ef30 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: dump: 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.656 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecdc0018c0 con 0x7fecec13ef30 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 1 -- 172.21.15.155:0/3112242728 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fecdc001c50 con 0x7fecec13ef30 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.674+0000 7feceaffd640 10 monclient: dump: 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:22.657 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.658 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.658 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:22.658 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:22.658 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:22.658 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:22.658 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:22.658 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.675+0000 7fecf2601640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.675+0000 7fecf2601640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.675+0000 7fecf2601640 4 set_mon_vals no callback set 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.676+0000 7fecf2601640 10 monclient: shutdown 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.676+0000 7fecf2601640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.676+0000 7fecf2601640 1 -- 172.21.15.155:0/3112242728 >> v1:172.21.15.183:6789/0 conn(0x7fecec13ef30 legacy=0x7fecec0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.676+0000 7fecf2601640 1 -- 172.21.15.155:0/3112242728 shutdown_connections 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.676+0000 7fecf2601640 1 -- 172.21.15.155:0/3112242728 >> 172.21.15.155:0/3112242728 conn(0x7fecec06b660 msgr2=0x7fecec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.678+0000 7fecf2601640 1 -- 172.21.15.155:0/3112242728 shutdown_connections 2024-03-29T11:01:22.659 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.678+0000 7fecf2601640 1 -- 172.21.15.155:0/3112242728 wait complete. 2024-03-29T11:01:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.679+0000 7fecf2601640 10 monclient: build_initial_monmap 2024-03-29T11:01:22.663 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.679+0000 7fecf2601640 10 monclient: monmap: 2024-03-29T11:01:22.663 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:22.663 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:22.663 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:22.663 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:22.663 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.679+0000 7fecf2601640 1 librados: starting msgr at 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.679+0000 7fecf2601640 1 librados: starting objecter 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.679+0000 7fecf2601640 1 Processor -- start 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.680+0000 7fecf2601640 1 -- start start 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.680+0000 7fecf2601640 1 librados: setting wanted keys 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.680+0000 7fecf2601640 1 librados: calling monclient init 2024-03-29T11:01:22.664 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.680+0000 7fecf2601640 10 monclient: init 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.680+0000 7fecf2601640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.680+0000 7fecf2601640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.680+0000 7fecf2601640 10 monclient(hunting): picked mon.noname-b con 0x7fecec0af8e0 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.681+0000 7fecf2601640 10 monclient(hunting): picked mon.noname-a con 0x7fecec0b0860 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.681+0000 7fecf2601640 10 monclient(hunting): picked mon.noname-c con 0x7fecec13ef30 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.681+0000 7fecf2601640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecec0b1dc0 con 0x7fecec0b0860 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.681+0000 7fecf2601640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecec133150 con 0x7fecec0af8e0 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.681+0000 7fecf2601640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecec0b3bd0 con 0x7fecec13ef30 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.681+0000 7fecf2601640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:22.665 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.681+0000 7fecf2601640 10 monclient(hunting): authenticate will time out at 1385.973941s 2024-03-29T11:01:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.686+0000 7fecf0b9e640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fecec13ef30 0x7fecec132a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:53843/0 (socket says 172.21.15.155:53843) 2024-03-29T11:01:22.667 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.687+0000 7fecf0b9e640 1 -- 172.21.15.155:0/3675167693 learned_addr learned my addr 172.21.15.155:0/3675167693 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 client.?.objecter ms_handle_connect 0x7fecec13ef30 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 client.?.objecter ms_handle_connect 0x7fecec0af8e0 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265781218 0 0) 0x7fecec0b3bd0 con 0x7fecec13ef30 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:22.672 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.690+0000 7fece8ff9640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:01:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feccc004eb0 con 0x7fecec13ef30 2024-03-29T11:01:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8331553 0 0) 0x7fecec133150 con 0x7fecec0af8e0 2024-03-29T11:01:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:01:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:01:22.673 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecec0b3bd0 con 0x7fecec0af8e0 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2721807240 0 0) 0x7fecec0b3bd0 con 0x7fecec0af8e0 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecec133150 con 0x7fecec0af8e0 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fece40019a0 con 0x7fecec0af8e0 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.691+0000 7fece8ff9640 10 monclient(hunting): dump: 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.674 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.692+0000 7fece8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:22.675 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.692+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 38949285 0 0) 0x7fecec133150 con 0x7fecec0af8e0 2024-03-29T11:01:22.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.692+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 >> v1:172.21.15.155:6790/0 conn(0x7fecec13ef30 legacy=0x7fecec132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:22.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.692+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 >> v1:172.21.15.155:6789/0 conn(0x7fecec0b0860 legacy=0x7fecec132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:22.676 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.692+0000 7fece8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:01:22.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.696+0000 7fece8ff9640 1 monclient: found mon.b 2024-03-29T11:01:22.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.696+0000 7fece8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:22.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.696+0000 7fece8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:22.677 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.696+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecec0b1850 con 0x7fecec0af8e0 2024-03-29T11:01:22.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.697+0000 7fece8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:01:22.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.697+0000 7fece8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:01:22.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.697+0000 7fece8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:22.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.697+0000 7fecf2601640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:01:22.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.697+0000 7fecf2601640 10 monclient: _renew_subs 2024-03-29T11:01:22.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.697+0000 7fecf2601640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:22.679 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.697+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecec0b1a40 con 0x7fecec0af8e0 2024-03-29T11:01:22.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.698+0000 7fecf2601640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:22.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.698+0000 7fecf2601640 10 monclient: _renew_subs 2024-03-29T11:01:22.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.698+0000 7fecf2601640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:22.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.698+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fece4006130 con 0x7fecec0af8e0 2024-03-29T11:01:22.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.698+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecec1d7070 con 0x7fecec0af8e0 2024-03-29T11:01:22.680 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.698+0000 7fecf2601640 1 librados: init done 2024-03-29T11:01:22.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.701+0000 7fece8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:22.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.701+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fece4006360 con 0x7fecec0af8e0 2024-03-29T11:01:22.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.701+0000 7fece8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:22.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.701+0000 7fece8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:22.681 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.701+0000 7fece8ff9640 10 monclient: dump: 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:22.682 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.701+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fece4001d60 con 0x7fecec0af8e0 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.702+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fece4084d60 con 0x7fecec0af8e0 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.702+0000 7fece8ff9640 10 client.4516.objecter ms_dispatch 0x7fecec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.702+0000 7fece8ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.702+0000 7fece8ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.702+0000 7fece8ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.702+0000 7feceb7fe640 10 monclient: get_auth_request con 0x7feccc06dba0 auth_method 0 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.703+0000 7fecea7fc640 4 set_mon_vals no callback set 2024-03-29T11:01:22.684 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.703+0000 7fece8ff9640 10 client.4516.objecter ms_handle_connect 0x7feccc06dba0 2024-03-29T11:01:22.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.704+0000 7fecf2601640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:22.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.704+0000 7fecf2601640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:22.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.704+0000 7fecf2601640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:22.685 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.704+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd0005080 con 0x7fecec0af8e0 2024-03-29T11:01:22.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.711+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fece40506c0 con 0x7fecec0af8e0 2024-03-29T11:01:22.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.711+0000 7fece8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:22.691 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:22.711+0000 7fece8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:23.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.040+0000 7fecf2601640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.023 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.040+0000 7fecf2601640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.040+0000 7fecf2601640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:23.024 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.040+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecd0005330 con 0x7fecec0af8e0 2024-03-29T11:01:23.030 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:23.030 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.043+0000 7fece8ff9640 1 -- 172.21.15.155:0/3675167693 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fece40020c0 con 0x7fecec0af8e0 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.043+0000 7fece8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.043+0000 7fece8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 10 librados: watch_flush enter 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 10 librados: watch_flush exit 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 >> v1:172.21.15.183:6816/1412921548 conn(0x7feccc06dba0 legacy=0x7feccc070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 10 monclient: shutdown 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 >> v1:172.21.15.183:6789/0 conn(0x7fecec0af8e0 legacy=0x7fecec0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:23.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 shutdown_connections 2024-03-29T11:01:23.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 >> 172.21.15.155:0/3675167693 conn(0x7fecec06b660 msgr2=0x7fecec13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:23.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.047+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 shutdown_connections 2024-03-29T11:01:23.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.048+0000 7fecf2601640 1 -- 172.21.15.155:0/3675167693 wait complete. 2024-03-29T11:01:23.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:23.048+0000 7fecf2601640 1 librados: shutdown 2024-03-29T11:01:23.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:29.853 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.040+0000 7f606ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.040+0000 7f606ffff640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.040+0000 7f606ffff640 10 monclient: monmap: 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:30.041944+0000 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:30.041944+0000 2024-03-29T11:01:30.021 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.022 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:30.022 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:30.022 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:30.022 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:30.022 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:30.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.041+0000 7f606ffff640 1 Processor -- start 2024-03-29T11:01:30.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.041+0000 7f606ffff640 1 -- start start 2024-03-29T11:01:30.025 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.041+0000 7f606ffff640 10 monclient: init 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.042+0000 7f606ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.042+0000 7f606ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.042+0000 7f606ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f60700a9750 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.042+0000 7f606ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f60700adca0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6070146a80 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606ffff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700abdc0 con 0x7f60700adca0 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606ffff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700b0070 con 0x7f6070146a80 2024-03-29T11:01:30.026 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606ffff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700af880 con 0x7f60700a9750 2024-03-29T11:01:30.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606ffff640 10 monclient(hunting): authenticate will time out at 1393.338656s 2024-03-29T11:01:30.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606f7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f6070146a80 0x7f6070151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:35567/0 (socket says 172.21.15.155:35567) 2024-03-29T11:01:30.027 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.045+0000 7f606f7fe640 1 -- 172.21.15.155:0/3437874141 learned_addr learned my addr 172.21.15.155:0/3437874141 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.046+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075353435 0 0) 0x7f60700b0070 con 0x7f6070146a80 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.046+0000 7f606e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.046+0000 7f606e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.046+0000 7f606e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.046+0000 7f606e7fc640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.046+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f605c003520 con 0x7f6070146a80 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.046+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1889171207 0 0) 0x7f605c003520 con 0x7f6070146a80 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.047+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 >> v1:172.21.15.155:6790/0 conn(0x7f60700a9750 legacy=0x7f6070143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.028 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.047+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 >> v1:172.21.15.155:6789/0 conn(0x7f60700adca0 legacy=0x7f6070146370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60700ac030 con 0x7f6070146a80 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.030 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606ffff640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6068001a10 con 0x7f6070146a80 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.049+0000 7f606e7fc640 10 monclient: dump: 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:30.031 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:30.032 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:30.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.050+0000 7f606ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:30.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.050+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6068001ca0 con 0x7f6070146a80 2024-03-29T11:01:30.032 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.050+0000 7f606e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.050+0000 7f606ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:30.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.050+0000 7f606ffff640 4 set_mon_vals no callback set 2024-03-29T11:01:30.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606ffff640 10 monclient: shutdown 2024-03-29T11:01:30.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:30.033 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606ffff640 1 -- 172.21.15.155:0/3437874141 >> v1:172.21.15.183:6789/0 conn(0x7f6070146a80 legacy=0x7f6070151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606e7fc640 1 -- 172.21.15.155:0/3437874141 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6068002030 con 0x7f6070146a80 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606e7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606ffff640 1 -- 172.21.15.155:0/3437874141 shutdown_connections 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606ffff640 1 -- 172.21.15.155:0/3437874141 >> 172.21.15.155:0/3437874141 conn(0x7f60700673e0 msgr2=0x7f60700b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606ffff640 1 -- 172.21.15.155:0/3437874141 shutdown_connections 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.051+0000 7f606ffff640 1 -- 172.21.15.155:0/3437874141 wait complete. 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.052+0000 7f606ffff640 10 monclient: build_initial_monmap 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.052+0000 7f606ffff640 10 monclient: monmap: 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:30.034 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:30.035 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:30.035 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:30.035 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:30.035 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:30.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.052+0000 7f606ffff640 1 librados: starting msgr at 2024-03-29T11:01:30.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.052+0000 7f606ffff640 1 librados: starting objecter 2024-03-29T11:01:30.035 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 1 Processor -- start 2024-03-29T11:01:30.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 1 -- start start 2024-03-29T11:01:30.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 1 librados: setting wanted keys 2024-03-29T11:01:30.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 1 librados: calling monclient init 2024-03-29T11:01:30.036 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 10 monclient: init 2024-03-29T11:01:30.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:30.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:30.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.053+0000 7f606ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f60700a9750 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:30.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.054+0000 7f606ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f60700adca0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:30.037 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.054+0000 7f606ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f6070146a80 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:30.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.054+0000 7f606ffff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607006d8c0 con 0x7f60700adca0 2024-03-29T11:01:30.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.054+0000 7f606ffff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700724a0 con 0x7f60700a9750 2024-03-29T11:01:30.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.054+0000 7f606ffff640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60700706d0 con 0x7f6070146a80 2024-03-29T11:01:30.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.054+0000 7f606ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:30.038 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.054+0000 7f606ffff640 10 monclient(hunting): authenticate will time out at 1393.346921s 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.057+0000 7f6067fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f60700adca0 0x7f6070070bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:45931/0 (socket says 172.21.15.155:45931) 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.057+0000 7f6067fff640 1 -- 172.21.15.155:0/4099576840 learned_addr learned my addr 172.21.15.155:0/4099576840 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.057+0000 7f60677fe640 10 client.?.objecter ms_handle_connect 0x7f60700adca0 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.057+0000 7f60677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 client.?.objecter ms_handle_connect 0x7f6070146a80 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822525066 0 0) 0x7f607006d8c0 con 0x7f60700adca0 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.040 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:01:30.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6050004eb0 con 0x7f60700adca0 2024-03-29T11:01:30.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220889066 0 0) 0x7f60700706d0 con 0x7f6070146a80 2024-03-29T11:01:30.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:30.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:30.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:30.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:01:30.041 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.058+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607006d8c0 con 0x7f6070146a80 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 850669269 0 0) 0x7f6050004eb0 con 0x7f60700adca0 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60700706d0 con 0x7f60700adca0 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 10 client.?.objecter ms_handle_connect 0x7f60700a9750 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6058001960 con 0x7f60700adca0 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 10 monclient(hunting): dump: 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:30.044 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.060+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2642640299 0 0) 0x7f60700706d0 con 0x7f60700adca0 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 >> v1:172.21.15.155:6790/0 conn(0x7f6070146a80 legacy=0x7f60701e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 >> v1:172.21.15.183:6789/0 conn(0x7f60700a9750 legacy=0x7f607013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 1 monclient: found mon.a 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:30.045 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60701df670 con 0x7f60700adca0 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f60677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 10 monclient: _renew_subs 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60701df920 con 0x7f60700adca0 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 10 monclient: _renew_subs 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f607013de20 con 0x7f60700adca0 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.061+0000 7f606ffff640 1 librados: init done 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.063+0000 7f606ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.063+0000 7f606ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.046 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.063+0000 7f606ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:30.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607004f730 con 0x7f60700adca0 2024-03-29T11:01:30.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6058001ac0 con 0x7f60700adca0 2024-03-29T11:01:30.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f60677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:30.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f60580038a0 con 0x7f60700adca0 2024-03-29T11:01:30.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f60677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:30.048 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f60677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f60677fe640 10 monclient: dump: 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.064+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f6058003ac0 con 0x7f60700adca0 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.065+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f6058084a40 con 0x7f60700adca0 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.065+0000 7f60677fe640 10 client.4545.objecter ms_dispatch 0x7f6070000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:30.049 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.065+0000 7f60677fe640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:30.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.065+0000 7f60677fe640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:30.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.065+0000 7f60677fe640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:01:30.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.065+0000 7f606d7fa640 4 set_mon_vals no callback set 2024-03-29T11:01:30.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.066+0000 7f606effd640 10 monclient: get_auth_request con 0x7f605006dce0 auth_method 0 2024-03-29T11:01:30.050 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.067+0000 7f60677fe640 10 client.4545.objecter ms_handle_connect 0x7f605006dce0 2024-03-29T11:01:30.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.075+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60580504a0 con 0x7f60700adca0 2024-03-29T11:01:30.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.075+0000 7f60677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:30.056 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.075+0000 7f60677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:30.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.450+0000 7f606ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:30.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.450+0000 7f606ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:30.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.450+0000 7f606ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:30.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.450+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6070140510 con 0x7f60700adca0 2024-03-29T11:01:30.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.457+0000 7f60677fe640 1 -- 172.21.15.155:0/4099576840 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f60580523d0 con 0x7f60700adca0 2024-03-29T11:01:30.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.457+0000 7f60677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:30.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.457+0000 7f60677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:30.439 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:30.439 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 10 librados: watch_flush enter 2024-03-29T11:01:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 10 librados: watch_flush exit 2024-03-29T11:01:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:01:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:01:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 >> v1:172.21.15.183:6816/1412921548 conn(0x7f605006dce0 legacy=0x7f60500701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 10 monclient: shutdown 2024-03-29T11:01:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.463+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 >> v1:172.21.15.155:6789/0 conn(0x7f60700adca0 legacy=0x7f6070070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.464+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 shutdown_connections 2024-03-29T11:01:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.464+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 >> 172.21.15.155:0/4099576840 conn(0x7f60700673e0 msgr2=0x7f60700aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.464+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 shutdown_connections 2024-03-29T11:01:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.464+0000 7f606ffff640 1 -- 172.21.15.155:0/4099576840 wait complete. 2024-03-29T11:01:30.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:30.464+0000 7f606ffff640 1 librados: shutdown 2024-03-29T11:01:30.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:37.270 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:37.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.430+0000 7f20e5458640 10 monclient: get_monmap_and_config 2024-03-29T11:01:37.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.430+0000 7f20e5458640 10 monclient: build_initial_monmap 2024-03-29T11:01:37.410 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.430+0000 7f20e5458640 10 monclient: monmap: 2024-03-29T11:01:37.410 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:37.431286+0000 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:37.431286+0000 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:37.411 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:37.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.431+0000 7f20e5458640 1 Processor -- start 2024-03-29T11:01:37.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.431+0000 7f20e5458640 1 -- start start 2024-03-29T11:01:37.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.431+0000 7f20e5458640 10 monclient: init 2024-03-29T11:01:37.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:37.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:37.412 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 10 monclient(hunting): picked mon.noname-c con 0x7f20e00b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 10 monclient(hunting): picked mon.noname-a con 0x7f20e0071eb0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 10 monclient(hunting): picked mon.noname-b con 0x7f20e0143d10 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e006ef80 con 0x7f20e0071eb0 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e006f260 con 0x7f20e00b1f60 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e00b40c0 con 0x7f20e0143d10 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.432+0000 7f20e5458640 10 monclient(hunting): authenticate will time out at 1400.724865s 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20df7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f20e0143d10 0x7f20e0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:54240/0 (socket says 172.21.15.155:54240) 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20df7fe640 1 -- 172.21.15.155:0/4051589606 learned_addr learned my addr 172.21.15.155:0/4051589606 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150083778 0 0) 0x7f20e006ef80 con 0x7f20e0071eb0 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:37.413 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:01:37.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c0003520 con 0x7f20e0071eb0 2024-03-29T11:01:37.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185725933 0 0) 0x7f20e00b40c0 con 0x7f20e0143d10 2024-03-29T11:01:37.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:37.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:37.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:37.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:01:37.414 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e006ef80 con 0x7f20e0143d10 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 758801191 0 0) 0x7f20c0003520 con 0x7f20e0071eb0 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 >> v1:172.21.15.155:6790/0 conn(0x7f20e0143d10 legacy=0x7f20e0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.433+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 >> v1:172.21.15.183:6789/0 conn(0x7f20e00b1f60 legacy=0x7f20e0143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.434+0000 7f20de7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 1 monclient: found mon.noname-a 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0155d70 con 0x7f20e0071eb0 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20c8001690 con 0x7f20e0071eb0 2024-03-29T11:01:37.415 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: dump: 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20e5458640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20e5458640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:37.416 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20c80017f0 con 0x7f20e0071eb0 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 1 -- 172.21.15.155:0/4051589606 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20c8001c00 con 0x7f20e0071eb0 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20de7fc640 10 monclient: dump: 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:37.417 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:37.418 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:37.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20e5458640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:37.418 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.435+0000 7f20e5458640 4 set_mon_vals no callback set 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.436+0000 7f20e5458640 10 monclient: shutdown 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.436+0000 7f20e5458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.436+0000 7f20e5458640 1 -- 172.21.15.155:0/4051589606 >> v1:172.21.15.155:6789/0 conn(0x7f20e0071eb0 legacy=0x7f20e0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.436+0000 7f20e5458640 1 -- 172.21.15.155:0/4051589606 shutdown_connections 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.436+0000 7f20e5458640 1 -- 172.21.15.155:0/4051589606 >> 172.21.15.155:0/4051589606 conn(0x7f20e0066a00 msgr2=0x7f20e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.437+0000 7f20e5458640 1 -- 172.21.15.155:0/4051589606 shutdown_connections 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.437+0000 7f20e5458640 1 -- 172.21.15.155:0/4051589606 wait complete. 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 10 monclient: build_initial_monmap 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 10 monclient: monmap: 2024-03-29T11:01:37.427 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 1 librados: starting msgr at 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 1 librados: starting objecter 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 1 Processor -- start 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 1 -- start start 2024-03-29T11:01:37.428 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 1 librados: setting wanted keys 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 1 librados: calling monclient init 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.438+0000 7f20e5458640 10 monclient: init 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20e5458640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20e5458640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20e5458640 10 monclient(hunting): picked mon.noname-b con 0x7f20e00b1f60 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20e5458640 10 monclient(hunting): picked mon.noname-c con 0x7f20e0143d10 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20d7fff640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f20e0143d10 0x7f20e013e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:54430/0 (socket says 172.21.15.155:54430) 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20df7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f20e0151fd0 0x7f20e013ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:39346/0 (socket says 172.21.15.155:39346) 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20d7fff640 1 -- 172.21.15.155:0/3481621705 learned_addr learned my addr 172.21.15.155:0/3481621705 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20e5458640 10 monclient(hunting): picked mon.noname-a con 0x7f20e0151fd0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20e5458640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e014b920 con 0x7f20e0151fd0 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20d77fe640 10 client.?.objecter ms_handle_connect 0x7f20e0151fd0 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.439+0000 7f20d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:37.429 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 client.?.objecter ms_handle_connect 0x7f20e0143d10 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20e5458640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e013f440 con 0x7f20e00b1f60 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20e5458640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e01deed0 con 0x7f20e0143d10 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159445687 0 0) 0x7f20e014b920 con 0x7f20e0151fd0 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20e5458640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20e5458640 10 monclient(hunting): authenticate will time out at 1400.732432s 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c4004eb0 con 0x7f20e0151fd0 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981762793 0 0) 0x7f20e01deed0 con 0x7f20e0143d10 2024-03-29T11:01:37.430 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.440+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e014b920 con 0x7f20e0143d10 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024122051 0 0) 0x7f20c4004eb0 con 0x7f20e0151fd0 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20e01deed0 con 0x7f20e0151fd0 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20d8001b60 con 0x7f20e0151fd0 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 10 monclient(hunting): dump: 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.431 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447847964 0 0) 0x7f20e014b920 con 0x7f20e0143d10 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20c4004eb0 con 0x7f20e0143d10 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20c8001bc0 con 0x7f20e0143d10 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 10 monclient(hunting): dump: 2024-03-29T11:01:37.432 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 643315617 0 0) 0x7f20e01deed0 con 0x7f20e0151fd0 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 >> v1:172.21.15.155:6790/0 conn(0x7f20e0143d10 legacy=0x7f20e013e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.441+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 >> v1:172.21.15.183:6789/0 conn(0x7f20e00b1f60 legacy=0x7f20e01df670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:37.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 1 monclient: found mon.a 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e01df170 con 0x7f20e0151fd0 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.442+0000 7f20d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 10 monclient: _renew_subs 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e01dfd80 con 0x7f20e0151fd0 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20d8002050 con 0x7f20e0151fd0 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:37.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f20d8004d00 con 0x7f20e0151fd0 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20ddffb640 4 set_mon_vals no callback set 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 10 monclient: _renew_subs 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e0070e80 con 0x7f20e0151fd0 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.443+0000 7f20e5458640 1 librados: init done 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.444+0000 7f20d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.444+0000 7f20d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.444+0000 7f20d77fe640 10 monclient: dump: 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:37.435 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f20d801acd0 con 0x7f20e0151fd0 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e0075f10 con 0x7f20e0151fd0 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f20d8085d10 con 0x7f20e0151fd0 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d77fe640 10 client.4551.objecter ms_dispatch 0x7f20e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d77fe640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d77fe640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:37.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.445+0000 7f20d77fe640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T11:01:37.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.446+0000 7f20deffd640 10 monclient: get_auth_request con 0x7f20c406de30 auth_method 0 2024-03-29T11:01:37.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.446+0000 7f20d77fe640 10 client.4551.objecter ms_handle_connect 0x7f20c406de30 2024-03-29T11:01:37.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.453+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20d80505c0 con 0x7f20e0151fd0 2024-03-29T11:01:37.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.453+0000 7f20d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:37.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.453+0000 7f20d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.771+0000 7f20d57fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.771+0000 7f20d57fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.771+0000 7f20d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.771+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20e00b6810 con 0x7f20e0151fd0 2024-03-29T11:01:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.772+0000 7f20d77fe640 1 -- 172.21.15.155:0/3481621705 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f20d80524f0 con 0x7f20e0151fd0 2024-03-29T11:01:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.772+0000 7f20d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:37.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.772+0000 7f20d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:37.754 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:37.754 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:37.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.775+0000 7f20d57fa640 10 librados: watch_flush enter 2024-03-29T11:01:37.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.775+0000 7f20d57fa640 10 librados: watch_flush exit 2024-03-29T11:01:37.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.775+0000 7f20d57fa640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T11:01:37.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.775+0000 7f20d57fa640 10 client.4551.objecter successfully canceled tick 2024-03-29T11:01:37.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.778+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 >> v1:172.21.15.183:6816/1412921548 conn(0x7f20c406de30 legacy=0x7f20c40702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:37.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.778+0000 7f20d57fa640 10 monclient: shutdown 2024-03-29T11:01:37.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.778+0000 7f20d57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:37.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.778+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 >> v1:172.21.15.155:6789/0 conn(0x7f20e0151fd0 legacy=0x7f20e013ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:37.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.778+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 shutdown_connections 2024-03-29T11:01:37.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.778+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 >> 172.21.15.155:0/3481621705 conn(0x7f20e0066a00 msgr2=0x7f20e0142a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:37.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.779+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 shutdown_connections 2024-03-29T11:01:37.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.779+0000 7f20d57fa640 1 -- 172.21.15.155:0/3481621705 wait complete. 2024-03-29T11:01:37.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:37.779+0000 7f20d57fa640 1 librados: shutdown 2024-03-29T11:01:37.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:44.584 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:44.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.755+0000 7f55c876d640 10 monclient: get_monmap_and_config 2024-03-29T11:01:44.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.755+0000 7f55c876d640 10 monclient: build_initial_monmap 2024-03-29T11:01:44.734 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.755+0000 7f55c876d640 10 monclient: monmap: 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:44.755724+0000 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:44.755724+0000 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:44.735 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:44.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.756+0000 7f55c876d640 1 Processor -- start 2024-03-29T11:01:44.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.756+0000 7f55c876d640 1 -- start start 2024-03-29T11:01:44.736 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.756+0000 7f55c876d640 10 monclient: init 2024-03-29T11:01:44.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:44.741 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 10 monclient(hunting): picked mon.noname-a con 0x7f55c00b4790 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 10 monclient(hunting): picked mon.noname-b con 0x7f55c00aea90 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 10 monclient(hunting): picked mon.noname-c con 0x7f55c00b39f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c00b0c60 con 0x7f55c00b4790 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c00af020 con 0x7f55c00b39f0 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c013fbd0 con 0x7f55c00aea90 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.757+0000 7f55c876d640 10 monclient(hunting): authenticate will time out at 1408.049684s 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5d08640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f55c00aea90 0x7f55c00af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39664/0 (socket says 172.21.15.155:39664) 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5d08640 1 -- 172.21.15.155:0/2216197649 learned_addr learned my addr 172.21.15.155:0/2216197649 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730604010 0 0) 0x7f55c013fbd0 con 0x7f55c00aea90 2024-03-29T11:01:44.742 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5507640 10 monclient(hunting): _negotiate 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5507640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5507640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5507640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.758+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ac003520 con 0x7f55c00aea90 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2705140786 0 0) 0x7f55ac003520 con 0x7f55c00aea90 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 >> v1:172.21.15.183:6789/0 conn(0x7f55c00b39f0 legacy=0x7f55c013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 >> v1:172.21.15.155:6789/0 conn(0x7f55c00b4790 legacy=0x7f55c00b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: _finish_hunting 0 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 1 monclient: found mon.noname-b 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c013fd60 con 0x7f55c00aea90 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: _finish_auth 0 2024-03-29T11:01:44.743 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: _check_auth_tickets 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c876d640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c876d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f55b0001690 con 0x7f55c00aea90 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: dump: 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:44.744 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c876d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 1 -- 172.21.15.155:0/2216197649 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55b00018b0 con 0x7f55c00aea90 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c5507640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c876d640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.759+0000 7f55c876d640 4 set_mon_vals no callback set 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.760+0000 7f55c876d640 10 monclient: shutdown 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.760+0000 7f55c876d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.760+0000 7f55c876d640 1 -- 172.21.15.155:0/2216197649 >> v1:172.21.15.155:6790/0 conn(0x7f55c00aea90 legacy=0x7f55c00af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.760+0000 7f55c876d640 1 -- 172.21.15.155:0/2216197649 shutdown_connections 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.760+0000 7f55c876d640 1 -- 172.21.15.155:0/2216197649 >> 172.21.15.155:0/2216197649 conn(0x7f55c0066a00 msgr2=0x7f55c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:44.745 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.760+0000 7f55c876d640 1 -- 172.21.15.155:0/2216197649 shutdown_connections 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.760+0000 7f55c876d640 1 -- 172.21.15.155:0/2216197649 wait complete. 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.761+0000 7f55c876d640 10 monclient: build_initial_monmap 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.761+0000 7f55c876d640 10 monclient: monmap: 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:44.746 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:44.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.762+0000 7f55c876d640 1 librados: starting msgr at 2024-03-29T11:01:44.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.762+0000 7f55c876d640 1 librados: starting objecter 2024-03-29T11:01:44.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.762+0000 7f55c876d640 1 Processor -- start 2024-03-29T11:01:44.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.762+0000 7f55c876d640 1 -- start start 2024-03-29T11:01:44.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.762+0000 7f55c876d640 1 librados: setting wanted keys 2024-03-29T11:01:44.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.762+0000 7f55c876d640 1 librados: calling monclient init 2024-03-29T11:01:44.747 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.762+0000 7f55c876d640 10 monclient: init 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 10 monclient(hunting): picked mon.noname-c con 0x7f55c00b39f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 10 monclient(hunting): picked mon.noname-a con 0x7f55c00b4790 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 10 monclient(hunting): picked mon.noname-b con 0x7f55c0070020 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c00a9680 con 0x7f55c00b4790 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c0070f00 con 0x7f55c0070020 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c6509640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f55c00b39f0 0x7f55c01db4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39890/0 (socket says 172.21.15.155:39890) 2024-03-29T11:01:44.751 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c6509640 1 -- 172.21.15.155:0/2907057624 learned_addr learned my addr 172.21.15.155:0/2907057624 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c01dbbe0 con 0x7f55c00b39f0 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55c876d640 10 monclient(hunting): authenticate will time out at 1408.055410s 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 client.?.objecter ms_handle_connect 0x7f55c00b39f0 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 client.?.objecter ms_handle_connect 0x7f55c00b4790 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115285847 0 0) 0x7f55c01dbbe0 con 0x7f55c00b39f0 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:01:44.752 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a4004eb0 con 0x7f55c00b39f0 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 client.?.objecter ms_handle_connect 0x7f55c0070020 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.763+0000 7f55b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251143205 0 0) 0x7f55a4004eb0 con 0x7f55c00b39f0 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c01dbbe0 con 0x7f55c00b39f0 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f55bc001930 con 0x7f55c00b39f0 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:44.753 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218977352 0 0) 0x7f55c0070f00 con 0x7f55c0070020 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a4004eb0 con 0x7f55c0070020 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 941818412 0 0) 0x7f55c01dbbe0 con 0x7f55c00b39f0 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 >> v1:172.21.15.183:6789/0 conn(0x7f55c0070020 legacy=0x7f55c01e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 >> v1:172.21.15.155:6789/0 conn(0x7f55c00b4790 legacy=0x7f55c006f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:44.754 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 monclient: found mon.c 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c01dbd70 con 0x7f55c00b39f0 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55c876d640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55c876d640 10 monclient: _renew_subs 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55c876d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c0071420 con 0x7f55c00b39f0 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55c876d640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.764+0000 7f55c876d640 10 monclient: _renew_subs 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.765+0000 7f55c876d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:44.755 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.765+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c0070650 con 0x7f55c00b39f0 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.765+0000 7f55c876d640 1 librados: init done 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.766+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55bc003610 con 0x7f55c00b39f0 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.766+0000 7f55b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.766+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f55bc0039e0 con 0x7f55c00b39f0 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.766+0000 7f55c4d06640 4 set_mon_vals no callback set 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.766+0000 7f55b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.766+0000 7f55b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.766+0000 7f55b6ffd640 10 monclient: dump: 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:44.756 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.767+0000 7f55c876d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.767+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f55bc003c00 con 0x7f55c00b39f0 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.767+0000 7f55c876d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.767+0000 7f55c876d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.767+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a8005080 con 0x7f55c00b39f0 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.767+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7f55bc084a00 con 0x7f55c00b39f0 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.768+0000 7f55b6ffd640 10 client.4628.objecter ms_dispatch 0x7f55c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.768+0000 7f55b6ffd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.768+0000 7f55b6ffd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.768+0000 7f55c5d08640 10 monclient: get_auth_request con 0x7f55a406dd90 auth_method 0 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.768+0000 7f55b6ffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:01:44.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.768+0000 7f55b6ffd640 10 client.4628.objecter ms_handle_connect 0x7f55a406dd90 2024-03-29T11:01:44.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.779+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55bc0502e0 con 0x7f55c00b39f0 2024-03-29T11:01:44.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.779+0000 7f55b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:44.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:44.779+0000 7f55b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.146+0000 7f55c876d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.146+0000 7f55c876d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.146+0000 7f55c876d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:01:45.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.146+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55a8005330 con 0x7f55c00b39f0 2024-03-29T11:01:45.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.148+0000 7f55b6ffd640 1 -- 172.21.15.155:0/2907057624 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7f55bc052210 con 0x7f55c00b39f0 2024-03-29T11:01:45.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.148+0000 7f55b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:45.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.148+0000 7f55b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:45.128 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:45.128 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 10 librados: watch_flush enter 2024-03-29T11:01:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 10 librados: watch_flush exit 2024-03-29T11:01:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:01:45.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:01:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 >> v1:172.21.15.183:6816/1412921548 conn(0x7f55a406dd90 legacy=0x7f55a4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 10 monclient: shutdown 2024-03-29T11:01:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 >> v1:172.21.15.155:6790/0 conn(0x7f55c00b39f0 legacy=0x7f55c01db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 shutdown_connections 2024-03-29T11:01:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 >> 172.21.15.155:0/2907057624 conn(0x7f55c0066a00 msgr2=0x7f55c00b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:45.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 shutdown_connections 2024-03-29T11:01:45.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 1 -- 172.21.15.155:0/2907057624 wait complete. 2024-03-29T11:01:45.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:45.152+0000 7f55c876d640 1 librados: shutdown 2024-03-29T11:01:45.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:51.952 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.116+0000 7fe8203db640 10 monclient: get_monmap_and_config 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.116+0000 7fe8203db640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.116+0000 7fe8203db640 10 monclient: monmap: 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:52.117264+0000 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:52.117264+0000 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.096 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:52.097 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:52.097 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:52.097 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.117+0000 7fe8203db640 1 Processor -- start 2024-03-29T11:01:52.101 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.117+0000 7fe8203db640 1 -- start start 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.117+0000 7fe8203db640 10 monclient: init 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 10 monclient(hunting): picked mon.noname-b con 0x7fe818070120 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 10 monclient(hunting): picked mon.noname-a con 0x7fe8180b0420 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 10 monclient(hunting): picked mon.noname-c con 0x7fe818070500 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818072090 con 0x7fe8180b0420 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818072370 con 0x7fe818070500 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818072650 con 0x7fe818070120 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe8203db640 10 monclient(hunting): authenticate will time out at 1415.410587s 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe81e978640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe818070500 0x7fe81814b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:58464/0 (socket says 172.21.15.155:58464) 2024-03-29T11:01:52.102 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.118+0000 7fe81e978640 1 -- 172.21.15.155:0/430736740 learned_addr learned my addr 172.21.15.155:0/430736740 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842580127 0 0) 0x7fe818072090 con 0x7fe8180b0420 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe804003520 con 0x7fe8180b0420 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974271835 0 0) 0x7fe818072370 con 0x7fe818070500 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.119+0000 7fe81d175640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818072090 con 0x7fe818070500 2024-03-29T11:01:52.103 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001216240 0 0) 0x7fe818072650 con 0x7fe818070120 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818072370 con 0x7fe818070120 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41231954 0 0) 0x7fe804003520 con 0x7fe8180b0420 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 >> v1:172.21.15.155:6790/0 conn(0x7fe818070120 legacy=0x7fe81813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 >> v1:172.21.15.183:6789/0 conn(0x7fe818070500 legacy=0x7fe81814b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 monclient: found mon.noname-a 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe818140090 con 0x7fe8180b0420 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.104 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe808001690 con 0x7fe8180b0420 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: dump: 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:52.105 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe8203db640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe8203db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8080018b0 con 0x7fe8180b0420 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe81d175640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe8203db640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.120+0000 7fe8203db640 4 set_mon_vals no callback set 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.121+0000 7fe81d175640 1 -- 172.21.15.155:0/430736740 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe808001bc0 con 0x7fe8180b0420 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.121+0000 7fe81d175640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.121+0000 7fe81d175640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.121+0000 7fe81d175640 10 monclient: dump: 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:52.106 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.121+0000 7fe8203db640 10 monclient: shutdown 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.121+0000 7fe8203db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.121+0000 7fe8203db640 1 -- 172.21.15.155:0/430736740 >> v1:172.21.15.155:6789/0 conn(0x7fe8180b0420 legacy=0x7fe8181405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.122+0000 7fe8203db640 1 -- 172.21.15.155:0/430736740 shutdown_connections 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.122+0000 7fe8203db640 1 -- 172.21.15.155:0/430736740 >> 172.21.15.155:0/430736740 conn(0x7fe818067500 msgr2=0x7fe8180b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:52.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.122+0000 7fe8203db640 1 -- 172.21.15.155:0/430736740 shutdown_connections 2024-03-29T11:01:52.108 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.122+0000 7fe8203db640 1 -- 172.21.15.155:0/430736740 wait complete. 2024-03-29T11:01:52.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 10 monclient: build_initial_monmap 2024-03-29T11:01:52.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 10 monclient: monmap: 2024-03-29T11:01:52.116 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 1 librados: starting msgr at 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 1 librados: starting objecter 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 1 Processor -- start 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 1 -- start start 2024-03-29T11:01:52.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 1 librados: setting wanted keys 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 1 librados: calling monclient init 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.123+0000 7fe8203db640 10 monclient: init 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.124+0000 7fe8203db640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.124+0000 7fe8203db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.124+0000 7fe8203db640 10 monclient(hunting): picked mon.noname-c con 0x7fe818070120 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.124+0000 7fe8203db640 10 monclient(hunting): picked mon.noname-a con 0x7fe818070500 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.124+0000 7fe8203db640 10 monclient(hunting): picked mon.noname-b con 0x7fe8180b0420 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.124+0000 7fe8203db640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180ac8b0 con 0x7fe818070500 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.124+0000 7fe8203db640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818125820 con 0x7fe8180b0420 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe8203db640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180acb00 con 0x7fe818070120 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe8203db640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe8203db640 10 monclient(hunting): authenticate will time out at 1415.416889s 2024-03-29T11:01:52.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe81e177640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fe818070120 0x7fe8180ab300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:60492/0 (socket says 172.21.15.155:60492) 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe81e177640 1 -- 172.21.15.155:0/925375727 learned_addr learned my addr 172.21.15.155:0/925375727 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe80effd640 10 client.?.objecter ms_handle_connect 0x7fe818070120 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe80effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe80effd640 10 client.?.objecter ms_handle_connect 0x7fe8180b0420 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe80effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe80effd640 10 client.?.objecter ms_handle_connect 0x7fe818070500 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe80effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.125+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605354547 0 0) 0x7fe818125820 con 0x7fe8180b0420 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc004eb0 con 0x7fe8180b0420 2024-03-29T11:01:52.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023618338 0 0) 0x7fe8180ac8b0 con 0x7fe818070500 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818125820 con 0x7fe818070500 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090269835 0 0) 0x7fe8180acb00 con 0x7fe818070120 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.126+0000 7fe80effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8180ac8b0 con 0x7fe818070120 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641812453 0 0) 0x7fe7fc004eb0 con 0x7fe8180b0420 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8180acb00 con 0x7fe8180b0420 2024-03-29T11:01:52.120 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe8140017e0 con 0x7fe8180b0420 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 10 monclient(hunting): dump: 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.121 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1424954313 0 0) 0x7fe8180ac8b0 con 0x7fe818070120 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7fc004eb0 con 0x7fe818070120 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1056236430 0 0) 0x7fe818125820 con 0x7fe818070500 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.127+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8180ac8b0 con 0x7fe818070500 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe810001c70 con 0x7fe818070120 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 10 monclient(hunting): dump: 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.122 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe8080013b0 con 0x7fe818070500 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 10 monclient(hunting): dump: 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.123 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2998541677 0 0) 0x7fe8180acb00 con 0x7fe8180b0420 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 >> v1:172.21.15.155:6790/0 conn(0x7fe818070120 legacy=0x7fe8180ab300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 >> v1:172.21.15.155:6789/0 conn(0x7fe818070500 legacy=0x7fe8181e1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 1 monclient: found mon.b 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.128+0000 7fe80effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe80effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8180aba10 con 0x7fe8180b0420 2024-03-29T11:01:52.124 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe80effd640 10 monclient: _finish_auth 0 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe80effd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe80effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 10 monclient: _renew_subs 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8180aa710 con 0x7fe8180b0420 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 10 monclient: _renew_subs 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8181e1e50 con 0x7fe8180b0420 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.129+0000 7fe8203db640 1 librados: init done 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.130+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe814001cb0 con 0x7fe8180b0420 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.131+0000 7fe80effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.131+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fe814004980 con 0x7fe8180b0420 2024-03-29T11:01:52.125 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.131+0000 7fe81c974640 4 set_mon_vals no callback set 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.131+0000 7fe8203db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.131+0000 7fe80effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.131+0000 7fe80effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.131+0000 7fe80effd640 10 monclient: dump: 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:52.126 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.132+0000 7fe8203db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.132+0000 7fe8203db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.132+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe800005080 con 0x7fe8180b0420 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.133+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fe814004ae0 con 0x7fe8180b0420 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.133+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3983287326 0 0) 0x7fe814084800 con 0x7fe8180b0420 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.134+0000 7fe80effd640 10 client.4531.objecter ms_dispatch 0x7fe818000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.134+0000 7fe80effd640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.134+0000 7fe80effd640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.134+0000 7fe80effd640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.135+0000 7fe81e177640 10 monclient: get_auth_request con 0x7fe7fc06e180 auth_method 0 2024-03-29T11:01:52.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.135+0000 7fe80effd640 10 client.4531.objecter ms_handle_connect 0x7fe7fc06e180 2024-03-29T11:01:52.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.141+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8140500e0 con 0x7fe8180b0420 2024-03-29T11:01:52.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.142+0000 7fe80effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:52.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.142+0000 7fe80effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:52.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.462+0000 7fe8203db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:52.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.462+0000 7fe8203db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:52.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.462+0000 7fe8203db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:01:52.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.463+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe800005330 con 0x7fe8180b0420 2024-03-29T11:01:52.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.464+0000 7fe80effd640 1 -- 172.21.15.155:0/925375727 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1205511617) 0x7fe814052010 con 0x7fe8180b0420 2024-03-29T11:01:52.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.464+0000 7fe80effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:52.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.464+0000 7fe80effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:52.443 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:52.444 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T10:51:55.345904+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:51:55.345904+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:52.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.467+0000 7fe8203db640 10 librados: watch_flush enter 2024-03-29T11:01:52.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.467+0000 7fe8203db640 10 librados: watch_flush exit 2024-03-29T11:01:52.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.467+0000 7fe8203db640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-29T11:01:52.447 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.468+0000 7fe8203db640 10 client.4531.objecter successfully canceled tick 2024-03-29T11:01:52.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.468+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 >> v1:172.21.15.183:6816/1412921548 conn(0x7fe7fc06e180 legacy=0x7fe7fc070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.468+0000 7fe8203db640 10 monclient: shutdown 2024-03-29T11:01:52.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.468+0000 7fe8203db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:52.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.468+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 >> v1:172.21.15.183:6789/0 conn(0x7fe8180b0420 legacy=0x7fe8181e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:52.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.469+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 shutdown_connections 2024-03-29T11:01:52.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.469+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 >> 172.21.15.155:0/925375727 conn(0x7fe818067500 msgr2=0x7fe81813c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:52.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.469+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 shutdown_connections 2024-03-29T11:01:52.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.469+0000 7fe8203db640 1 -- 172.21.15.155:0/925375727 wait complete. 2024-03-29T11:01:52.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:52.469+0000 7fe8203db640 1 librados: shutdown 2024-03-29T11:01:52.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:59.270 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.440+0000 7f22f8953640 10 monclient: get_monmap_and_config 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.440+0000 7f22f8953640 10 monclient: build_initial_monmap 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.440+0000 7f22f8953640 10 monclient: monmap: 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:01:59.441102+0000 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:01:59.441102+0000 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:01:59.420 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:01:59.421 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:59.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.441+0000 7f22f8953640 1 Processor -- start 2024-03-29T11:01:59.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.441+0000 7f22f8953640 1 -- start start 2024-03-29T11:01:59.421 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.441+0000 7f22f8953640 10 monclient: init 2024-03-29T11:01:59.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.442+0000 7f22f8953640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:59.422 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.442+0000 7f22f8953640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.444+0000 7f22f8953640 10 monclient(hunting): picked mon.noname-b con 0x7f22ec038c60 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.446+0000 7f22f8953640 10 monclient(hunting): picked mon.noname-c con 0x7f22ec0324b0 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.446+0000 7f22f259c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f22ec0324b0 0x7f22ec0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:34353/0 (socket says 172.21.15.155:34353) 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.446+0000 7f22f259c640 1 -- 172.21.15.155:0/3639189449 learned_addr learned my addr 172.21.15.155:0/3639189449 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.446+0000 7f22f8953640 10 monclient(hunting): picked mon.noname-a con 0x7f22ec0d11e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.446+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0caf80 con 0x7f22ec0d11e0 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec039850 con 0x7f22ec0324b0 2024-03-29T11:01:59.431 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0c78d0 con 0x7f22ec038c60 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f8953640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f8953640 10 monclient(hunting): authenticate will time out at 1422.738998s 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699050749 0 0) 0x7f22ec039850 con 0x7f22ec0324b0 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f1d9b640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.447+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22dc003520 con 0x7f22ec0324b0 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483329453 0 0) 0x7f22ec0caf80 con 0x7f22ec0d11e0 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:59.432 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec039850 con 0x7f22ec0d11e0 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 713561937 0 0) 0x7f22dc003520 con 0x7f22ec0324b0 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 >> v1:172.21.15.155:6790/0 conn(0x7f22ec038c60 legacy=0x7f22ec0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 >> v1:172.21.15.155:6789/0 conn(0x7f22ec0d11e0 legacy=0x7f22ec0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 1 monclient: found mon.noname-c 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.448+0000 7f22f1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec0c7a60 con 0x7f22ec0324b0 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f22e8001610 con 0x7f22ec0324b0 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f8953640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:01:59.433 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f8953640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 10 monclient: dump: 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:59.434 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e80018a0 con 0x7f22ec0324b0 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f8953640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f1d9b640 1 -- 172.21.15.155:0/3639189449 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f22e8001c70 con 0x7f22ec0324b0 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f8953640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.449+0000 7f22f8953640 4 set_mon_vals no callback set 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.450+0000 7f22f8953640 10 monclient: shutdown 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.450+0000 7f22f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.451+0000 7f22f1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.451+0000 7f22f1d9b640 10 monclient: dump: 2024-03-29T11:01:59.435 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.451+0000 7f22f8953640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:59.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.451+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 >> v1:172.21.15.183:6789/0 conn(0x7f22ec0324b0 legacy=0x7f22ec0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.453+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 shutdown_connections 2024-03-29T11:01:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.453+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 >> 172.21.15.155:0/3639189449 conn(0x7f22ec005c20 msgr2=0x7f22ec006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.453+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 shutdown_connections 2024-03-29T11:01:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.454+0000 7f22f8953640 1 -- 172.21.15.155:0/3639189449 wait complete. 2024-03-29T11:01:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.455+0000 7f22f8953640 10 monclient: build_initial_monmap 2024-03-29T11:01:59.438 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.455+0000 7f22f8953640 10 monclient: monmap: 2024-03-29T11:01:59.438 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.456+0000 7f22f8953640 1 librados: starting msgr at 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.456+0000 7f22f8953640 1 librados: starting objecter 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.456+0000 7f22f8953640 1 Processor -- start 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.456+0000 7f22f8953640 1 -- start start 2024-03-29T11:01:59.439 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.456+0000 7f22f8953640 1 librados: setting wanted keys 2024-03-29T11:01:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.456+0000 7f22f8953640 1 librados: calling monclient init 2024-03-29T11:01:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.456+0000 7f22f8953640 10 monclient: init 2024-03-29T11:01:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.457+0000 7f22f8953640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.457+0000 7f22f8953640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:59.440 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.458+0000 7f22f8953640 10 monclient(hunting): picked mon.noname-a con 0x7f22ec0324b0 addr v1:172.21.15.155:6789/0 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f8953640 10 monclient(hunting): picked mon.noname-b con 0x7f22ec038c60 addr v1:172.21.15.183:6789/0 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f8953640 10 monclient(hunting): picked mon.noname-c con 0x7f22ec0d11e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f8953640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0bcc30 con 0x7f22ec0324b0 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f8953640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0c4900 con 0x7f22ec038c60 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f8953640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0c4e20 con 0x7f22ec0d11e0 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f8953640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f8953640 10 monclient(hunting): authenticate will time out at 1422.751315s 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f2d9d640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f22ec0324b0 0x7f22ec15fee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:47014/0 (socket says 172.21.15.155:47014) 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22f2d9d640 1 -- 172.21.15.155:0/2256415371 learned_addr learned my addr 172.21.15.155:0/2256415371 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 10 client.?.objecter ms_handle_connect 0x7f22ec0324b0 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838069616 0 0) 0x7f22ec0bcc30 con 0x7f22ec0324b0 2024-03-29T11:01:59.441 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.459+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e0004eb0 con 0x7f22ec0324b0 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.461+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4292126158 0 0) 0x7f22e0004eb0 con 0x7f22ec0324b0 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.461+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ec0bcc30 con 0x7f22ec0324b0 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.461+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f22e4004080 con 0x7f22ec0324b0 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.461+0000 7f22db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.461+0000 7f22db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.461+0000 7f22db7fe640 10 monclient(hunting): dump: 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:59.442 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.461+0000 7f22db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.462+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3652601292 0 0) 0x7f22ec0bcc30 con 0x7f22ec0324b0 2024-03-29T11:01:59.443 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.462+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 >> v1:172.21.15.155:6790/0 conn(0x7f22ec0d11e0 legacy=0x7f22ec0bc520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:59.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 >> v1:172.21.15.183:6789/0 conn(0x7f22ec038c60 legacy=0x7f22ec0bbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:59.445 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 1 monclient: found mon.a 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec0c38b0 con 0x7f22ec0324b0 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.465+0000 7f22db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22f8953640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:01:59.446 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e40043a0 con 0x7f22ec0324b0 2024-03-29T11:01:59.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:59.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f22e4001320 con 0x7f22ec0324b0 2024-03-29T11:01:59.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:59.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:59.448 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22db7fe640 10 monclient: dump: 2024-03-29T11:01:59.448 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.466+0000 7f22f159a640 4 set_mon_vals no callback set 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 10 monclient: _renew_subs 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec0c9c10 con 0x7f22ec0324b0 2024-03-29T11:01:59.449 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 10 monclient: _renew_subs 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec1607d0 con 0x7f22ec0324b0 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.467+0000 7f22f8953640 1 librados: init done 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.469+0000 7f22f8953640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.469+0000 7f22f8953640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.469+0000 7f22f8953640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:59.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.469+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec007b90 con 0x7f22ec0324b0 2024-03-29T11:01:59.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.470+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f22e4001480 con 0x7f22ec0324b0 2024-03-29T11:01:59.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.470+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7f22e4084b40 con 0x7f22ec0324b0 2024-03-29T11:01:59.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.470+0000 7f22db7fe640 10 client.4566.objecter ms_dispatch 0x7f22f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:01:59.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.470+0000 7f22db7fe640 3 client.4566.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:01:59.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.470+0000 7f22db7fe640 3 client.4566.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:01:59.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.470+0000 7f22db7fe640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T11:01:59.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.472+0000 7f22f259c640 10 monclient: get_auth_request con 0x7f22e006dc90 auth_method 0 2024-03-29T11:01:59.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.472+0000 7f22db7fe640 10 client.4566.objecter ms_handle_connect 0x7f22e006dc90 2024-03-29T11:01:59.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.480+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22e4050420 con 0x7f22ec0324b0 2024-03-29T11:01:59.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.480+0000 7f22db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:59.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.480+0000 7f22db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:59.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.801+0000 7f22f8953640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:59.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.801+0000 7f22f8953640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:59.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.801+0000 7f22f8953640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:01:59.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.801+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22ec0c9fb0 con 0x7f22ec0324b0 2024-03-29T11:01:59.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.802+0000 7f22db7fe640 1 -- 172.21.15.155:0/2256415371 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7f22e4052350 con 0x7f22ec0324b0 2024-03-29T11:01:59.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.802+0000 7f22db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:59.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.802+0000 7f22db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:59.783 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:01:59.783 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 10 librados: watch_flush enter 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 10 librados: watch_flush exit 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 10 client.4566.objecter successfully canceled tick 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 >> v1:172.21.15.183:6816/1412921548 conn(0x7f22e006dc90 legacy=0x7f22e0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 10 monclient: shutdown 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 >> v1:172.21.15.155:6789/0 conn(0x7f22ec0324b0 legacy=0x7f22ec15fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 shutdown_connections 2024-03-29T11:01:59.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 >> 172.21.15.155:0/2256415371 conn(0x7f22ec005c20 msgr2=0x7f22ec03bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:59.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 shutdown_connections 2024-03-29T11:01:59.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.806+0000 7f22f8953640 1 -- 172.21.15.155:0/2256415371 wait complete. 2024-03-29T11:01:59.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:01:59.807+0000 7f22f8953640 1 librados: shutdown 2024-03-29T11:01:59.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:06.613 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:06.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.785+0000 7fcd97a2e640 10 monclient: get_monmap_and_config 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.785+0000 7fcd97a2e640 10 monclient: build_initial_monmap 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.785+0000 7fcd97a2e640 10 monclient: monmap: 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:02:06.786495+0000 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:02:06.786495+0000 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.787+0000 7fcd97a2e640 1 Processor -- start 2024-03-29T11:02:06.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.787+0000 7fcd97a2e640 1 -- start start 2024-03-29T11:02:06.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.787+0000 7fcd97a2e640 10 monclient: init 2024-03-29T11:02:06.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.788+0000 7fcd97a2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:06.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.788+0000 7fcd97a2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:06.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.788+0000 7fcd97a2e640 10 monclient(hunting): picked mon.noname-a con 0x7fcd900a94e0 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.788+0000 7fcd97a2e640 10 monclient(hunting): picked mon.noname-b con 0x7fcd900a98c0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.788+0000 7fcd97a2e640 10 monclient(hunting): picked mon.noname-c con 0x7fcd900ad470 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.788+0000 7fcd97a2e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900ac210 con 0x7fcd900a94e0 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.788+0000 7fcd97a2e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900ac4f0 con 0x7fcd900ad470 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.790+0000 7fcd94fc9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fcd900a98c0 0x7fcd901525d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:43418/0 (socket says 172.21.15.155:43418) 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.790+0000 7fcd94fc9640 1 -- 172.21.15.155:0/205137931 learned_addr learned my addr 172.21.15.155:0/205137931 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.790+0000 7fcd97a2e640 1 -- 172.21.15.155:0/205137931 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900ab280 con 0x7fcd900a98c0 2024-03-29T11:02:06.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.790+0000 7fcd97a2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:06.771 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.790+0000 7fcd97a2e640 10 monclient(hunting): authenticate will time out at 1430.082358s 2024-03-29T11:02:06.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427475112 0 0) 0x7fcd900ab280 con 0x7fcd900a98c0 2024-03-29T11:02:06.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:06.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:06.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:06.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:02:06.772 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c005730 con 0x7fcd900a98c0 2024-03-29T11:02:06.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797551358 0 0) 0x7fcd900ac210 con 0x7fcd900a94e0 2024-03-29T11:02:06.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:06.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:06.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:06.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:02:06.773 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.792+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd900ab280 con 0x7fcd900a94e0 2024-03-29T11:02:06.775 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 790632411 0 0) 0x7fcd7c005730 con 0x7fcd900a98c0 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 >> v1:172.21.15.183:6789/0 conn(0x7fcd900ad470 legacy=0x7fcd90152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 >> v1:172.21.15.155:6789/0 conn(0x7fcd900a94e0 legacy=0x7fcd900acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 1 monclient: found mon.noname-b 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd900b00c0 con 0x7fcd900a98c0 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 10 monclient: _finish_auth 0 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcd88001250 con 0x7fcd900a98c0 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:06.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd87fff640 10 monclient: dump: 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:06.777 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd97a2e640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.795+0000 7fcd97a2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd880013b0 con 0x7fcd900a98c0 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd87fff640 1 -- 172.21.15.155:0/205137931 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcd88001800 con 0x7fcd900a98c0 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd87fff640 10 monclient: dump: 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.778 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd97a2e640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:06.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.796+0000 7fcd97a2e640 4 set_mon_vals no callback set 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.797+0000 7fcd97a2e640 10 monclient: shutdown 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.797+0000 7fcd97a2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.797+0000 7fcd97a2e640 1 -- 172.21.15.155:0/205137931 >> v1:172.21.15.155:6790/0 conn(0x7fcd900a98c0 legacy=0x7fcd901525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.797+0000 7fcd97a2e640 1 -- 172.21.15.155:0/205137931 shutdown_connections 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.797+0000 7fcd97a2e640 1 -- 172.21.15.155:0/205137931 >> 172.21.15.155:0/205137931 conn(0x7fcd900b3a00 msgr2=0x7fcd900b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.798+0000 7fcd97a2e640 1 -- 172.21.15.155:0/205137931 shutdown_connections 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.798+0000 7fcd97a2e640 1 -- 172.21.15.155:0/205137931 wait complete. 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 10 monclient: build_initial_monmap 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 10 monclient: monmap: 2024-03-29T11:02:06.781 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 1 librados: starting msgr at 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 1 librados: starting objecter 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 1 Processor -- start 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 1 -- start start 2024-03-29T11:02:06.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 1 librados: setting wanted keys 2024-03-29T11:02:06.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 1 librados: calling monclient init 2024-03-29T11:02:06.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.799+0000 7fcd97a2e640 10 monclient: init 2024-03-29T11:02:06.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.800+0000 7fcd97a2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:06.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.800+0000 7fcd97a2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:06.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.801+0000 7fcd97a2e640 10 monclient(hunting): picked mon.noname-c con 0x7fcd900a94e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:06.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.801+0000 7fcd97a2e640 10 monclient(hunting): picked mon.noname-a con 0x7fcd900ad470 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:06.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.801+0000 7fcd97a2e640 10 monclient(hunting): picked mon.noname-b con 0x7fcd90152d10 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:06.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.803+0000 7fcd97a2e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd901e1230 con 0x7fcd900ad470 2024-03-29T11:02:06.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.803+0000 7fcd97a2e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd90153180 con 0x7fcd90152d10 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.803+0000 7fcd97a2e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9014cf60 con 0x7fcd900a94e0 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.803+0000 7fcd97a2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.803+0000 7fcd97a2e640 10 monclient(hunting): authenticate will time out at 1430.095455s 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.801+0000 7fcd957ca640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fcd900a94e0 0x7fcd9008a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:47524/0 (socket says 172.21.15.155:47524) 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.803+0000 7fcd957ca640 1 -- 172.21.15.155:0/4110381555 learned_addr learned my addr 172.21.15.155:0/4110381555 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 client.?.objecter ms_handle_connect 0x7fcd900a94e0 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691964118 0 0) 0x7fcd9014cf60 con 0x7fcd900a94e0 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd74004eb0 con 0x7fcd900a94e0 2024-03-29T11:02:06.786 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715555520 0 0) 0x7fcd74004eb0 con 0x7fcd900a94e0 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd9014cf60 con 0x7fcd900a94e0 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcd8c001a20 con 0x7fcd900a94e0 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): dump: 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:06.787 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.804+0000 7fcd85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2973544995 0 0) 0x7fcd9014cf60 con 0x7fcd900a94e0 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 >> v1:172.21.15.183:6789/0 conn(0x7fcd90152d10 legacy=0x7fcd901ef2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 >> v1:172.21.15.155:6789/0 conn(0x7fcd900ad470 legacy=0x7fcd901e3a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 1 monclient: found mon.c 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9014d0f0 con 0x7fcd900a94e0 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:02:06.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 10 monclient: _renew_subs 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9008acb0 con 0x7fcd900a94e0 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd8c001d00 con 0x7fcd900a94e0 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fcd8c003a40 con 0x7fcd900a94e0 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd85ffb640 10 monclient: dump: 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:06.789 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 10 monclient: _renew_subs 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd901e4170 con 0x7fcd900a94e0 2024-03-29T11:02:06.790 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.805+0000 7fcd877fe640 4 set_mon_vals no callback set 2024-03-29T11:02:06.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.807+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fcd8c003ba0 con 0x7fcd900a94e0 2024-03-29T11:02:06.791 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.808+0000 7fcd94fc9640 10 monclient: get_auth_request con 0x7fcd7406dc00 auth_method 0 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.808+0000 7fcd97a2e640 1 librados: init done 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.808+0000 7fcd85ffb640 10 client.4640.objecter ms_handle_connect 0x7fcd7406dc00 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.810+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7fcd8c054980 con 0x7fcd900a94e0 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.810+0000 7fcd97a2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.810+0000 7fcd97a2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.810+0000 7fcd97a2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.810+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9004f730 con 0x7fcd900a94e0 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.810+0000 7fcd85ffb640 10 client.4640.objecter ms_dispatch 0x7fcd90000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.810+0000 7fcd85ffb640 3 client.4640.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.811+0000 7fcd85ffb640 3 client.4640.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:06.792 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.811+0000 7fcd85ffb640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:02:06.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.821+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcd8c0502c0 con 0x7fcd900a94e0 2024-03-29T11:02:06.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.821+0000 7fcd85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:06.800 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:06.821+0000 7fcd85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.146+0000 7fcd97a2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.146+0000 7fcd97a2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.126 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.146+0000 7fcd97a2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:07.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.146+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd90074e10 con 0x7fcd900a94e0 2024-03-29T11:02:07.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.148+0000 7fcd85ffb640 1 -- 172.21.15.155:0/4110381555 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7fcd8c0523c0 con 0x7fcd900a94e0 2024-03-29T11:02:07.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.148+0000 7fcd85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:07.127 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.148+0000 7fcd85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:07.128 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:02:07.128 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:07.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.152+0000 7fcd97a2e640 10 librados: watch_flush enter 2024-03-29T11:02:07.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.152+0000 7fcd97a2e640 10 librados: watch_flush exit 2024-03-29T11:02:07.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.152+0000 7fcd97a2e640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:02:07.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.152+0000 7fcd97a2e640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:02:07.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.152+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 >> v1:172.21.15.183:6816/1412921548 conn(0x7fcd7406dc00 legacy=0x7fcd740700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.152+0000 7fcd97a2e640 10 monclient: shutdown 2024-03-29T11:02:07.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.152+0000 7fcd97a2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:07.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.153+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 >> v1:172.21.15.155:6790/0 conn(0x7fcd900a94e0 legacy=0x7fcd9008a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:07.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.153+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 shutdown_connections 2024-03-29T11:02:07.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.153+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 >> 172.21.15.155:0/4110381555 conn(0x7fcd900b3a00 msgr2=0x7fcd900af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.155+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 shutdown_connections 2024-03-29T11:02:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.155+0000 7fcd97a2e640 1 -- 172.21.15.155:0/4110381555 wait complete. 2024-03-29T11:02:07.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:07.155+0000 7fcd97a2e640 1 librados: shutdown 2024-03-29T11:02:07.151 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:13.958 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:14.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.125+0000 7f71426b5640 10 monclient: get_monmap_and_config 2024-03-29T11:02:14.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.125+0000 7f71426b5640 10 monclient: build_initial_monmap 2024-03-29T11:02:14.107 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.125+0000 7f71426b5640 10 monclient: monmap: 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:02:14.126778+0000 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:02:14.126778+0000 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:02:14.108 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:02:14.109 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:14.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.126+0000 7f71426b5640 1 Processor -- start 2024-03-29T11:02:14.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.126+0000 7f71426b5640 1 -- start start 2024-03-29T11:02:14.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.126+0000 7f71426b5640 10 monclient: init 2024-03-29T11:02:14.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:14.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:14.109 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 10 monclient(hunting): picked mon.noname-c con 0x7f713c0ad450 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:14.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 10 monclient(hunting): picked mon.noname-a con 0x7f713c0ad830 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:14.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 10 monclient(hunting): picked mon.noname-b con 0x7f713c157020 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:14.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0aab80 con 0x7f713c0ad830 2024-03-29T11:02:14.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0aae60 con 0x7f713c0ad450 2024-03-29T11:02:14.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0a94e0 con 0x7f713c157020 2024-03-29T11:02:14.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:14.110 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.127+0000 7f71426b5640 10 monclient(hunting): authenticate will time out at 1437.420576s 2024-03-29T11:02:14.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.130+0000 7f7140c52640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f713c157020 0x7f713c163440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:36880/0 (socket says 172.21.15.155:36880) 2024-03-29T11:02:14.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.130+0000 7f7140c52640 1 -- 172.21.15.155:0/2693498284 learned_addr learned my addr 172.21.15.155:0/2693498284 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:14.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679730820 0 0) 0x7f713c0a94e0 con 0x7f713c157020 2024-03-29T11:02:14.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.115 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7124003520 con 0x7f713c157020 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022305237 0 0) 0x7f7124003520 con 0x7f713c157020 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 >> v1:172.21.15.183:6789/0 conn(0x7f713c0ad450 legacy=0x7f713c156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 >> v1:172.21.15.155:6789/0 conn(0x7f713c0ad830 legacy=0x7f713c0ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 monclient: found mon.noname-b 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0af7c0 con 0x7f713c157020 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient: _finish_auth 0 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f71426b5640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:02:14.116 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f71426b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7130001690 con 0x7f713c157020 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f713affd640 10 monclient: dump: 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:14.117 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.131+0000 7f71426b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7130001860 con 0x7f713c157020 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f713affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f71426b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f71426b5640 4 set_mon_vals no callback set 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f713affd640 1 -- 172.21.15.155:0/2693498284 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7130001c70 con 0x7f713c157020 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f713affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f713affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.132+0000 7f713affd640 10 monclient: dump: 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.118 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.133+0000 7f71426b5640 10 monclient: shutdown 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.133+0000 7f71426b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.133+0000 7f71426b5640 1 -- 172.21.15.155:0/2693498284 >> v1:172.21.15.155:6790/0 conn(0x7f713c157020 legacy=0x7f713c163440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.133+0000 7f71426b5640 1 -- 172.21.15.155:0/2693498284 shutdown_connections 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.133+0000 7f71426b5640 1 -- 172.21.15.155:0/2693498284 >> 172.21.15.155:0/2693498284 conn(0x7f713c0b35e0 msgr2=0x7f713c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.134+0000 7f71426b5640 1 -- 172.21.15.155:0/2693498284 shutdown_connections 2024-03-29T11:02:14.119 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.134+0000 7f71426b5640 1 -- 172.21.15.155:0/2693498284 wait complete. 2024-03-29T11:02:14.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 10 monclient: build_initial_monmap 2024-03-29T11:02:14.128 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 10 monclient: monmap: 2024-03-29T11:02:14.128 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:14.128 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:14.128 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:02:14.128 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:02:14.128 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 1 librados: starting msgr at 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 1 librados: starting objecter 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 1 Processor -- start 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 1 -- start start 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 1 librados: setting wanted keys 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 1 librados: calling monclient init 2024-03-29T11:02:14.129 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.135+0000 7f71426b5640 10 monclient: init 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 10 monclient(hunting): picked mon.noname-a con 0x7f713c1f0210 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 10 monclient(hunting): picked mon.noname-c con 0x7f713c06e7a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 10 monclient(hunting): picked mon.noname-b con 0x7f713c07aff0 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c06fd90 con 0x7f713c1f0210 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c150210 con 0x7f713c07aff0 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c07da00 con 0x7f713c06e7a0 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.136+0000 7f71426b5640 10 monclient(hunting): authenticate will time out at 1437.429712s 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.137+0000 7f713b7fe640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f713c06e7a0 0x7f713c14f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:37320/0 (socket says 172.21.15.155:37320) 2024-03-29T11:02:14.130 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.137+0000 7f713b7fe640 1 -- 172.21.15.155:0/290156395 learned_addr learned my addr 172.21.15.155:0/290156395 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.137+0000 7f7138ff9640 10 client.?.objecter ms_handle_connect 0x7f713c06e7a0 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.137+0000 7f7138ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.138+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924065246 0 0) 0x7f713c07da00 con 0x7f713c06e7a0 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.138+0000 7f7138ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.138+0000 7f7138ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.138+0000 7f7138ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.138+0000 7f7138ff9640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.138+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7128004eb0 con 0x7f713c06e7a0 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2407037742 0 0) 0x7f7128004eb0 con 0x7f713c06e7a0 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7128005300 con 0x7f713c06e7a0 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7134001d70 con 0x7f713c06e7a0 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.131 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:14.132 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 848976428 0 0) 0x7f7128005300 con 0x7f713c06e7a0 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.139+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 >> v1:172.21.15.183:6789/0 conn(0x7f713c07aff0 legacy=0x7f713c14fb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 >> v1:172.21.15.155:6789/0 conn(0x7f713c1f0210 legacy=0x7f713c1f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 1 monclient: found mon.c 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:14.133 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c070000 con 0x7f713c06e7a0 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 10 monclient: _finish_auth 0 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 10 client.?.objecter ms_handle_connect 0x7f713c1f0210 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f7138ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f71426b5640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f71426b5640 10 monclient: _renew_subs 2024-03-29T11:02:14.134 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f71426b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.140+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c1f0d00 con 0x7f713c06e7a0 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.141+0000 7f71426b5640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.141+0000 7f71426b5640 10 monclient: _renew_subs 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.141+0000 7f71426b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.141+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c1f95b0 con 0x7f713c06e7a0 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.141+0000 7f71426b5640 1 librados: init done 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.142+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7134002070 con 0x7f713c06e7a0 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.142+0000 7f7138ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:14.135 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.142+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f7134001d70 con 0x7f713c06e7a0 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.142+0000 7f7138ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.142+0000 7f7138ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.142+0000 7f7138ff9640 10 monclient: dump: 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:14.136 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.142+0000 7f713a7fc640 4 set_mon_vals no callback set 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.143+0000 7f71426b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.145+0000 7f71426b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.145+0000 7f71426b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:14.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.145+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711c005080 con 0x7f713c06e7a0 2024-03-29T11:02:14.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.146+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f7134003c40 con 0x7f713c06e7a0 2024-03-29T11:02:14.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.146+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7f71340549e0 con 0x7f713c06e7a0 2024-03-29T11:02:14.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.146+0000 7f7138ff9640 10 client.4646.objecter ms_dispatch 0x7f713c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:14.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.146+0000 7f7138ff9640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:14.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.146+0000 7f7138ff9640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:14.138 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.147+0000 7f7138ff9640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:02:14.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.147+0000 7f713bfff640 10 monclient: get_auth_request con 0x7f712806dfe0 auth_method 0 2024-03-29T11:02:14.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.149+0000 7f7138ff9640 10 client.4646.objecter ms_handle_connect 0x7f712806dfe0 2024-03-29T11:02:14.139 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.154+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7134050c80 con 0x7f713c06e7a0 2024-03-29T11:02:14.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.154+0000 7f7138ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:14.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.154+0000 7f7138ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:14.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.477+0000 7f71426b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:14.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.477+0000 7f71426b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:14.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.477+0000 7f71426b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:14.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.477+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f711c005330 con 0x7f713c06e7a0 2024-03-29T11:02:14.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.478+0000 7f7138ff9640 1 -- 172.21.15.155:0/290156395 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7f7134050c80 con 0x7f713c06e7a0 2024-03-29T11:02:14.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.478+0000 7f7138ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:14.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.478+0000 7f7138ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:14.460 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:02:14.460 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:14.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.482+0000 7f71426b5640 10 librados: watch_flush enter 2024-03-29T11:02:14.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.482+0000 7f71426b5640 10 librados: watch_flush exit 2024-03-29T11:02:14.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.482+0000 7f71426b5640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:02:14.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.482+0000 7f71426b5640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:02:14.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.482+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 >> v1:172.21.15.183:6816/1412921548 conn(0x7f712806dfe0 legacy=0x7f71280704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.483+0000 7f71426b5640 10 monclient: shutdown 2024-03-29T11:02:14.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.483+0000 7f71426b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:14.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.483+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 >> v1:172.21.15.155:6790/0 conn(0x7f713c06e7a0 legacy=0x7f713c14f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:14.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.483+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 shutdown_connections 2024-03-29T11:02:14.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.483+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 >> 172.21.15.155:0/290156395 conn(0x7f713c0b35e0 msgr2=0x7f713c0b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:14.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.484+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 shutdown_connections 2024-03-29T11:02:14.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.484+0000 7f71426b5640 1 -- 172.21.15.155:0/290156395 wait complete. 2024-03-29T11:02:14.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:14.485+0000 7f71426b5640 1 librados: shutdown 2024-03-29T11:02:14.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:21.289 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.453+0000 7f9c0344e640 10 monclient: get_monmap_and_config 2024-03-29T11:02:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.453+0000 7f9c0344e640 10 monclient: build_initial_monmap 2024-03-29T11:02:21.435 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.453+0000 7f9c0344e640 10 monclient: monmap: 2024-03-29T11:02:21.435 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:21.435 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:02:21.454723+0000 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:02:21.454723+0000 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:21.436 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.454+0000 7f9c0344e640 1 Processor -- start 2024-03-29T11:02:21.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.454+0000 7f9c0344e640 1 -- start start 2024-03-29T11:02:21.437 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.454+0000 7f9c0344e640 10 monclient: init 2024-03-29T11:02:21.450 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 10 monclient(hunting): picked mon.noname-a con 0x7f9bfc0b0770 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 10 monclient(hunting): picked mon.noname-c con 0x7f9bfc14a700 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 10 monclient(hunting): picked mon.noname-b con 0x7f9bfc0ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0aba70 con 0x7f9bfc0b0770 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0aecb0 con 0x7f9bfc14a700 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0af150 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.451 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c0344e640 10 monclient(hunting): authenticate will time out at 1444.748484s 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c011ea640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f9bfc0b0770 0x7f9bfc149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:41404/0 (socket says 172.21.15.155:41404) 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.455+0000 7f9c011ea640 1 -- 172.21.15.155:0/2007857806 learned_addr learned my addr 172.21.15.155:0/2007857806 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604842787 0 0) 0x7f9bfc0af150 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be8003520 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994643638 0 0) 0x7f9bfc0aba70 con 0x7f9bfc0b0770 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.452 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc0af150 con 0x7f9bfc0b0770 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3099319183 0 0) 0x7f9bfc0af150 con 0x7f9bfc0b0770 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 >> v1:172.21.15.155:6790/0 conn(0x7f9bfc0ac5a0 legacy=0x7f9bfc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.456+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 >> v1:172.21.15.183:6789/0 conn(0x7f9bfc14a700 legacy=0x7f9bfc0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.458+0000 7f9bf3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.458+0000 7f9bf3fff640 1 monclient: found mon.noname-a 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.458+0000 7f9bf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc0ac9d0 con 0x7f9bfc0b0770 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: _finish_auth 0 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:21.453 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9c0344e640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9c0344e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9bf40019f0 con 0x7f9bfc0b0770 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: dump: 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:21.454 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9c0344e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf4001c10 con 0x7f9bfc0b0770 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 1 -- 172.21.15.155:0/2007857806 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9bf4001fa0 con 0x7f9bfc0b0770 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9bf3fff640 10 monclient: dump: 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:21.455 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9c0344e640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.459+0000 7f9c0344e640 4 set_mon_vals no callback set 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.460+0000 7f9c0344e640 10 monclient: shutdown 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.460+0000 7f9c0344e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.460+0000 7f9c0344e640 1 -- 172.21.15.155:0/2007857806 >> v1:172.21.15.155:6789/0 conn(0x7f9bfc0b0770 legacy=0x7f9bfc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.460+0000 7f9c0344e640 1 -- 172.21.15.155:0/2007857806 shutdown_connections 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.460+0000 7f9c0344e640 1 -- 172.21.15.155:0/2007857806 >> 172.21.15.155:0/2007857806 conn(0x7f9bfc06b660 msgr2=0x7f9bfc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.460+0000 7f9c0344e640 1 -- 172.21.15.155:0/2007857806 shutdown_connections 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.460+0000 7f9c0344e640 1 -- 172.21.15.155:0/2007857806 wait complete. 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.461+0000 7f9c0344e640 10 monclient: build_initial_monmap 2024-03-29T11:02:21.456 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.461+0000 7f9c0344e640 10 monclient: monmap: 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.462+0000 7f9c0344e640 1 librados: starting msgr at 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.462+0000 7f9c0344e640 1 librados: starting objecter 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.462+0000 7f9c0344e640 1 Processor -- start 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.462+0000 7f9c0344e640 1 -- start start 2024-03-29T11:02:21.457 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.462+0000 7f9c0344e640 1 librados: setting wanted keys 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.462+0000 7f9c0344e640 1 librados: calling monclient init 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.462+0000 7f9c0344e640 10 monclient: init 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 10 monclient(hunting): picked mon.noname-c con 0x7f9bfc0ac5a0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 10 monclient(hunting): picked mon.noname-b con 0x7f9bfc0b0770 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 10 monclient(hunting): picked mon.noname-a con 0x7f9bfc14a700 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc1e44e0 con 0x7f9bfc14a700 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc1225c0 con 0x7f9bfc0b0770 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc1233f0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c011ea640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f9bfc0ac5a0 0x7f9bfc122ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:33006/0 (socket says 172.21.15.155:33006) 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c011ea640 1 -- 172.21.15.155:0/1913001173 learned_addr learned my addr 172.21.15.155:0/1913001173 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 client.?.objecter ms_handle_connect 0x7f9bfc0ac5a0 2024-03-29T11:02:21.458 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 client.?.objecter ms_handle_connect 0x7f9bfc14a700 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9c0344e640 10 monclient(hunting): authenticate will time out at 1444.756494s 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986188010 0 0) 0x7f9bfc1233f0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be0004eb0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413006985 0 0) 0x7f9bfc1e44e0 con 0x7f9bfc14a700 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:21.459 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.463+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc1233f0 con 0x7f9bfc14a700 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276035863 0 0) 0x7f9be0004eb0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9be00064f0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9bf40019f0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 10 monclient(hunting): dump: 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:21.460 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 192704900 0 0) 0x7f9bfc1233f0 con 0x7f9bfc14a700 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9be0006ba0 con 0x7f9bfc14a700 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9bec0018d0 con 0x7f9bfc14a700 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 10 monclient(hunting): dump: 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.461 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1792603239 0 0) 0x7f9be00064f0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.464+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 >> v1:172.21.15.183:6789/0 conn(0x7f9bfc0b0770 legacy=0x7f9bfc11fef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 >> v1:172.21.15.155:6789/0 conn(0x7f9bfc14a700 legacy=0x7f9bfc120600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 1 monclient: found mon.c 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:21.462 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc1e3e30 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9bf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 10 monclient: _renew_subs 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bfc1e4080 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 10 monclient: _renew_subs 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bfc1e4b50 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.465+0000 7f9c0344e640 1 librados: init done 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.467+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf4004e10 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.467+0000 7f9bf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:21.463 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.467+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f9bf40019f0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.467+0000 7f9bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.467+0000 7f9bf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.467+0000 7f9bf1ffb640 10 monclient: dump: 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.467+0000 7f9bf37fe640 4 set_mon_vals no callback set 2024-03-29T11:02:21.464 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.468+0000 7f9c0344e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.468+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f9bf4005100 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.468+0000 7f9c0344e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.468+0000 7f9c0344e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.468+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc005080 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.469+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7f9bf4084ca0 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.469+0000 7f9bf1ffb640 10 client.4652.objecter ms_dispatch 0x7f9bfc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.469+0000 7f9bf1ffb640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.469+0000 7f9bf1ffb640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.469+0000 7f9c009e9640 10 monclient: get_auth_request con 0x7f9be006e360 auth_method 0 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.469+0000 7f9bf1ffb640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T11:02:21.465 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.469+0000 7f9bf1ffb640 10 client.4652.objecter ms_handle_connect 0x7f9be006e360 2024-03-29T11:02:21.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.475+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9bf4050720 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.475+0000 7f9bf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:21.466 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.475+0000 7f9bf1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:21.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.794+0000 7f9c0344e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:21.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.794+0000 7f9c0344e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:21.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.794+0000 7f9c0344e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:21.774 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.794+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bdc005330 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.796+0000 7f9bf1ffb640 1 -- 172.21.15.155:0/1913001173 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7f9bf4050720 con 0x7f9bfc0ac5a0 2024-03-29T11:02:21.776 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.796+0000 7f9bf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:21.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.796+0000 7f9bf1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:21.778 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:02:21.778 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:21.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.801+0000 7f9c0344e640 10 librados: watch_flush enter 2024-03-29T11:02:21.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.802+0000 7f9c0344e640 10 librados: watch_flush exit 2024-03-29T11:02:21.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.802+0000 7f9c0344e640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T11:02:21.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.802+0000 7f9c0344e640 10 client.4652.objecter successfully canceled tick 2024-03-29T11:02:21.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.802+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 >> v1:172.21.15.183:6816/1412921548 conn(0x7f9be006e360 legacy=0x7f9be0070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 10 monclient: shutdown 2024-03-29T11:02:21.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:21.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 >> v1:172.21.15.155:6790/0 conn(0x7f9bfc0ac5a0 legacy=0x7f9bfc122ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:21.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 shutdown_connections 2024-03-29T11:02:21.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 >> 172.21.15.155:0/1913001173 conn(0x7f9bfc06b660 msgr2=0x7f9bfc0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:21.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 shutdown_connections 2024-03-29T11:02:21.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 1 -- 172.21.15.155:0/1913001173 wait complete. 2024-03-29T11:02:21.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:21.803+0000 7f9c0344e640 1 librados: shutdown 2024-03-29T11:02:21.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:28.608 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:28.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.778+0000 7f62844a8640 10 monclient: get_monmap_and_config 2024-03-29T11:02:28.757 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.778+0000 7f62844a8640 10 monclient: build_initial_monmap 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.778+0000 7f62844a8640 10 monclient: monmap: 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:02:28.778697+0000 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:02:28.778697+0000 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:02:28.758 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:28.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.779+0000 7f62844a8640 1 Processor -- start 2024-03-29T11:02:28.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.779+0000 7f62844a8640 1 -- start start 2024-03-29T11:02:28.759 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.779+0000 7f62844a8640 10 monclient: init 2024-03-29T11:02:28.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f62844a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:28.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f62844a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:28.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f62844a8640 10 monclient(hunting): picked mon.noname-a con 0x7f627c1457f0 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:28.760 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f62844a8640 10 monclient(hunting): picked mon.noname-c con 0x7f627c13f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:28.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f62844a8640 10 monclient(hunting): picked mon.noname-b con 0x7f627c0a9d70 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:28.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f62844a8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c0aaef0 con 0x7f627c1457f0 2024-03-29T11:02:28.761 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f6282a45640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7f627c0a9d70 0x7f627c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41608/0 (socket says 172.21.15.155:41608) 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f62844a8640 1 -- 172.21.15.155:0/4257000574 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c145bd0 con 0x7f627c13f650 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f62844a8640 1 -- 172.21.15.155:0/4257000574 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c145eb0 con 0x7f627c0a9d70 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f62844a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f62844a8640 10 monclient(hunting): authenticate will time out at 1452.073921s 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.781+0000 7f6282a45640 1 -- 172.21.15.155:0/4257000574 learned_addr learned my addr 172.21.15.155:0/4257000574 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256319934 0 0) 0x7f627c145eb0 con 0x7f627c0a9d70 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _negotiate 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:28.762 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:02:28.763 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6264003520 con 0x7f627c0a9d70 2024-03-29T11:02:28.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27264988 0 0) 0x7f627c145bd0 con 0x7f627c13f650 2024-03-29T11:02:28.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _negotiate 2024-03-29T11:02:28.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:28.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:28.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:02:28.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f627c145eb0 con 0x7f627c13f650 2024-03-29T11:02:28.765 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781092262 0 0) 0x7f627c0aaef0 con 0x7f627c1457f0 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _negotiate 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.782+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f627c145bd0 con 0x7f627c1457f0 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377852008 0 0) 0x7f6264003520 con 0x7f627c0a9d70 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 >> v1:172.21.15.183:6789/0 conn(0x7f627c13f650 legacy=0x7f627c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 >> v1:172.21.15.155:6789/0 conn(0x7f627c1457f0 legacy=0x7f627c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 10 monclient: _finish_hunting 0 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 1 monclient: found mon.noname-b 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c146340 con 0x7f627c0a9d70 2024-03-29T11:02:28.766 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 10 monclient: _finish_auth 0 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 10 monclient: _check_auth_tickets 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f62844a8640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f62844a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6274001a10 con 0x7f627c0a9d70 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 10 monclient: dump: 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:28.767 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f62844a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.783+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6274001bf0 con 0x7f627c0a9d70 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f6281242640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f62844a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f62844a8640 4 set_mon_vals no callback set 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f6281242640 1 -- 172.21.15.155:0/4257000574 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6274001a10 con 0x7f627c0a9d70 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f6281242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f6281242640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:28.768 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f6281242640 10 monclient: dump: 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.784+0000 7f62844a8640 10 monclient: shutdown 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.785+0000 7f62844a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.785+0000 7f62844a8640 1 -- 172.21.15.155:0/4257000574 >> v1:172.21.15.155:6790/0 conn(0x7f627c0a9d70 legacy=0x7f627c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.785+0000 7f62844a8640 1 -- 172.21.15.155:0/4257000574 shutdown_connections 2024-03-29T11:02:28.769 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.785+0000 7f62844a8640 1 -- 172.21.15.155:0/4257000574 >> 172.21.15.155:0/4257000574 conn(0x7f627c066a00 msgr2=0x7f627c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:28.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.785+0000 7f62844a8640 1 -- 172.21.15.155:0/4257000574 shutdown_connections 2024-03-29T11:02:28.770 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.785+0000 7f62844a8640 1 -- 172.21.15.155:0/4257000574 wait complete. 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.786+0000 7f62844a8640 10 monclient: build_initial_monmap 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.786+0000 7f62844a8640 10 monclient: monmap: 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:28.777 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.787+0000 7f62844a8640 1 librados: starting msgr at 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.787+0000 7f62844a8640 1 librados: starting objecter 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.787+0000 7f62844a8640 1 Processor -- start 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.787+0000 7f62844a8640 1 -- start start 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.787+0000 7f62844a8640 1 librados: setting wanted keys 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.787+0000 7f62844a8640 1 librados: calling monclient init 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.787+0000 7f62844a8640 10 monclient: init 2024-03-29T11:02:28.778 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.788+0000 7f62844a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.788+0000 7f62844a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.788+0000 7f62844a8640 10 monclient(hunting): picked mon.noname-b con 0x7f627c0a9d70 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.788+0000 7f62844a8640 10 monclient(hunting): picked mon.noname-a con 0x7f627c13f650 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.788+0000 7f62844a8640 10 monclient(hunting): picked mon.noname-c con 0x7f627c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f62844a8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c138e80 con 0x7f627c13f650 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f6282244640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f627c0a9d70 0x7f627c13b680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:36112/0 (socket says 172.21.15.155:36112) 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f6282244640 1 -- 172.21.15.155:0/2747254578 learned_addr learned my addr 172.21.15.155:0/2747254578 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f62844a8640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c13ac80 con 0x7f627c0a9d70 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f62844a8640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c13aef0 con 0x7f627c1457f0 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f62844a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 client.?.objecter ms_handle_connect 0x7f627c13f650 2024-03-29T11:02:28.779 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 client.?.objecter ms_handle_connect 0x7f627c0a9d70 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f62844a8640 10 monclient(hunting): authenticate will time out at 1452.081188s 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 client.?.objecter ms_handle_connect 0x7f627c1457f0 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051156232 0 0) 0x7f627c138e80 con 0x7f627c13f650 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:28.780 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.789+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6268004eb0 con 0x7f627c13f650 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942783138 0 0) 0x7f6268004eb0 con 0x7f627c13f650 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f627c138e80 con 0x7f627c13f650 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f62700018b0 con 0x7f627c13f650 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): dump: 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:28.781 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443742400 0 0) 0x7f627c13aef0 con 0x7f627c1457f0 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:02:28.782 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6268004eb0 con 0x7f627c1457f0 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 333008320 0 0) 0x7f627c138e80 con 0x7f627c13f650 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 >> v1:172.21.15.155:6790/0 conn(0x7f627c1457f0 legacy=0x7f627c1e1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 >> v1:172.21.15.183:6789/0 conn(0x7f627c0a9d70 legacy=0x7f627c13b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 monclient: found mon.a 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c139ae0 con 0x7f627c13f650 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient: _finish_auth 0 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:28.783 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f627affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 10 monclient: _renew_subs 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627c142970 con 0x7f627c13f650 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 10 monclient: _renew_subs 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f627c1e3430 con 0x7f627c13f650 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.790+0000 7f62844a8640 1 librados: init done 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.792+0000 7f6278ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:28.784 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.792+0000 7f6278ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.792+0000 7f6278ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.792+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c005080 con 0x7f627c13f650 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.793+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6270003610 con 0x7f627c13f650 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.793+0000 7f627affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.793+0000 7f6280a41640 4 set_mon_vals no callback set 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.793+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7f6270003920 con 0x7f627c13f650 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.793+0000 7f627affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.793+0000 7f627affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.794+0000 7f627affd640 10 monclient: dump: 2024-03-29T11:02:28.785 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:28.786 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.795+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7f62700198f0 con 0x7f627c13f650 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.796+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7f6270083850 con 0x7f627c13f650 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.796+0000 7f627affd640 10 client.4587.objecter ms_dispatch 0x7f627c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.796+0000 7f627affd640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.796+0000 7f627affd640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.796+0000 7f627affd640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.796+0000 7f6282244640 10 monclient: get_auth_request con 0x7f626806ddf0 auth_method 0 2024-03-29T11:02:28.787 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.797+0000 7f627affd640 10 client.4587.objecter ms_handle_connect 0x7f626806ddf0 2024-03-29T11:02:28.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.802+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f627004ef60 con 0x7f627c13f650 2024-03-29T11:02:28.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.802+0000 7f627affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:28.788 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:28.802+0000 7f627affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:29.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.160+0000 7f6278ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:29.140 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.160+0000 7f6278ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:29.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.160+0000 7f6278ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.155:6789/0 2024-03-29T11:02:29.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.160+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 --> v1:172.21.15.155:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f625c005330 con 0x7f627c13f650 2024-03-29T11:02:29.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.161+0000 7f627affd640 1 -- 172.21.15.155:0/2747254578 <== mon.0 v1:172.21.15.155:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7f6270051060 con 0x7f627c13f650 2024-03-29T11:02:29.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.161+0000 7f627affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:29.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.161+0000 7f627affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:29.142 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:02:29.142 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:29.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 10 librados: watch_flush enter 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 10 librados: watch_flush exit 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 >> v1:172.21.15.183:6816/1412921548 conn(0x7f626806ddf0 legacy=0x7f62680702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 10 monclient: shutdown 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 >> v1:172.21.15.155:6789/0 conn(0x7f627c13f650 legacy=0x7f627c1393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 shutdown_connections 2024-03-29T11:02:29.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.166+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 >> 172.21.15.155:0/2747254578 conn(0x7f627c066a00 msgr2=0x7f627c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:29.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.167+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 shutdown_connections 2024-03-29T11:02:29.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.167+0000 7f6278ff9640 1 -- 172.21.15.155:0/2747254578 wait complete. 2024-03-29T11:02:29.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:29.167+0000 7f6278ff9640 1 librados: shutdown 2024-03-29T11:02:29.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:35.967 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.157+0000 7fc8a92ae640 10 monclient: get_monmap_and_config 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.157+0000 7fc8a92ae640 10 monclient: build_initial_monmap 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.157+0000 7fc8a92ae640 10 monclient: monmap: 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:02:36.158548+0000 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:02:36.158548+0000 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:36.137 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:36.138 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:36.138 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:02:36.138 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:02:36.138 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:36.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.158+0000 7fc8a92ae640 1 Processor -- start 2024-03-29T11:02:36.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 1 -- start start 2024-03-29T11:02:36.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient: init 2024-03-29T11:02:36.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:36.141 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a41457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a413f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a40a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a40aaef0 con 0x7fc8a40a9d70 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a4145bd0 con 0x7fc8a413f650 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a4145eb0 con 0x7fc8a41457f0 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.159+0000 7fc8a92ae640 10 monclient(hunting): authenticate will time out at 1459.451846s 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.160+0000 7fc8a2ffd640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc8a41457f0 0x7fc8a413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:39846/0 (socket says 172.21.15.155:39846) 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.160+0000 7fc8a2ffd640 1 -- 172.21.15.155:0/3437453373 learned_addr learned my addr 172.21.15.155:0/3437453373 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618162746 0 0) 0x7fc8a4145eb0 con 0x7fc8a41457f0 2024-03-29T11:02:36.142 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:02:36.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc890003520 con 0x7fc8a41457f0 2024-03-29T11:02:36.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727691296 0 0) 0x7fc890003520 con 0x7fc8a41457f0 2024-03-29T11:02:36.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 >> v1:172.21.15.183:6789/0 conn(0x7fc8a413f650 legacy=0x7fc8a40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:36.143 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.161+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 >> v1:172.21.15.155:6789/0 conn(0x7fc8a40a9d70 legacy=0x7fc8a414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:36.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:36.145 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a4146340 con 0x7fc8a41457f0 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc89c001690 con 0x7fc8a41457f0 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.165+0000 7fc8a1ffb640 10 monclient: dump: 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.146 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a92ae640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:02:36.147 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a92ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc89c001860 con 0x7fc8a41457f0 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a1ffb640 1 -- 172.21.15.155:0/3437453373 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc89c001cb0 con 0x7fc8a41457f0 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.166+0000 7fc8a1ffb640 10 monclient: dump: 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.148 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.167+0000 7fc8a92ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.167+0000 7fc8a92ae640 4 set_mon_vals no callback set 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.168+0000 7fc8a92ae640 10 monclient: shutdown 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.168+0000 7fc8a92ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.168+0000 7fc8a92ae640 1 -- 172.21.15.155:0/3437453373 >> v1:172.21.15.155:6790/0 conn(0x7fc8a41457f0 legacy=0x7fc8a413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.168+0000 7fc8a92ae640 1 -- 172.21.15.155:0/3437453373 shutdown_connections 2024-03-29T11:02:36.149 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.168+0000 7fc8a92ae640 1 -- 172.21.15.155:0/3437453373 >> 172.21.15.155:0/3437453373 conn(0x7fc8a4066a00 msgr2=0x7fc8a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:36.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.170+0000 7fc8a92ae640 1 -- 172.21.15.155:0/3437453373 shutdown_connections 2024-03-29T11:02:36.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.170+0000 7fc8a92ae640 1 -- 172.21.15.155:0/3437453373 wait complete. 2024-03-29T11:02:36.150 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.171+0000 7fc8a92ae640 10 monclient: build_initial_monmap 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.171+0000 7fc8a92ae640 10 monclient: monmap: 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:36.151 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.171+0000 7fc8a92ae640 1 librados: starting msgr at 2024-03-29T11:02:36.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.171+0000 7fc8a92ae640 1 librados: starting objecter 2024-03-29T11:02:36.152 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.171+0000 7fc8a92ae640 1 Processor -- start 2024-03-29T11:02:36.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 1 -- start start 2024-03-29T11:02:36.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 1 librados: setting wanted keys 2024-03-29T11:02:36.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 1 librados: calling monclient init 2024-03-29T11:02:36.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 10 monclient: init 2024-03-29T11:02:36.153 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc8a40a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc8a413f650 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc8a41457f0 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a4136b80 con 0x7fc8a40a9d70 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a4136e60 con 0x7fc8a41457f0 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a4137140 con 0x7fc8a413f650 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.172+0000 7fc8a92ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc8a92ae640 10 monclient(hunting): authenticate will time out at 1459.464879s 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc8a27fc640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc8a413f650 0x7fc8a406ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:41974/0 (socket says 172.21.15.155:41974) 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc8a27fc640 1 -- 172.21.15.155:0/219019504 learned_addr learned my addr 172.21.15.155:0/219019504 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 10 client.?.objecter ms_handle_connect 0x7fc8a413f650 2024-03-29T11:02:36.154 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:36.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796825234 0 0) 0x7fc8a4137140 con 0x7fc8a413f650 2024-03-29T11:02:36.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:36.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:36.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:36.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:02:36.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.173+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc884004eb0 con 0x7fc8a413f650 2024-03-29T11:02:36.155 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3524995769 0 0) 0x7fc884004eb0 con 0x7fc8a413f650 2024-03-29T11:02:36.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8a4137140 con 0x7fc8a413f650 2024-03-29T11:02:36.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc894001bf0 con 0x7fc8a413f650 2024-03-29T11:02:36.156 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 10 monclient(hunting): dump: 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3022540888 0 0) 0x7fc8a4137140 con 0x7fc8a413f650 2024-03-29T11:02:36.157 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.174+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 >> v1:172.21.15.183:6789/0 conn(0x7fc8a41457f0 legacy=0x7fc8a41e6190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:36.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 >> v1:172.21.15.155:6789/0 conn(0x7fc8a40a9d70 legacy=0x7fc8a406e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:36.159 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 1 monclient: found mon.c 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a4071410 con 0x7fc8a413f650 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 10 client.?.objecter ms_handle_connect 0x7fc8a40a9d70 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc89b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc8a92ae640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc8a92ae640 10 monclient: _renew_subs 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc8a92ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.178+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a4070d10 con 0x7fc8a413f650 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.179+0000 7fc8a92ae640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:36.160 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.179+0000 7fc8a92ae640 10 monclient: _renew_subs 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.179+0000 7fc8a92ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.179+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a41e78b0 con 0x7fc8a413f650 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.179+0000 7fc8a92ae640 1 librados: init done 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.180+0000 7fc8a92ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.180+0000 7fc8a92ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.180+0000 7fc8a92ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.180+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a4071ec0 con 0x7fc8a413f650 2024-03-29T11:02:36.161 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.181+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc894001d50 con 0x7fc8a413f650 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.181+0000 7fc89b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.181+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc894004e40 con 0x7fc8a413f650 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.181+0000 7fc89b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.181+0000 7fc89b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.181+0000 7fc89b7fe640 10 monclient: dump: 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:36.162 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:36.163 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:36.163 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.181+0000 7fc8a17fa640 4 set_mon_vals no callback set 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.183+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fc89401ae10 con 0x7fc8a413f650 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.184+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7fc894084e00 con 0x7fc8a413f650 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.184+0000 7fc89b7fe640 10 client.4664.objecter ms_dispatch 0x7fc8a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.184+0000 7fc89b7fe640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.184+0000 7fc89b7fe640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.184+0000 7fc89b7fe640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.184+0000 7fc8a2ffd640 10 monclient: get_auth_request con 0x7fc88406dd20 auth_method 0 2024-03-29T11:02:36.164 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.185+0000 7fc89b7fe640 10 client.4664.objecter ms_handle_connect 0x7fc88406dd20 2024-03-29T11:02:36.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.193+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc894050510 con 0x7fc8a413f650 2024-03-29T11:02:36.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.193+0000 7fc89b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:36.172 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.193+0000 7fc89b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:36.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.552+0000 7fc8a92ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:36.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.552+0000 7fc8a92ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:36.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.552+0000 7fc8a92ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:36.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.552+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8a4075f10 con 0x7fc8a413f650 2024-03-29T11:02:36.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.553+0000 7fc89b7fe640 1 -- 172.21.15.155:0/219019504 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7fc894052610 con 0x7fc8a413f650 2024-03-29T11:02:36.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.553+0000 7fc89b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:36.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.553+0000 7fc89b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:36.534 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:02:36.534 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:36.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.558+0000 7fc8a92ae640 10 librados: watch_flush enter 2024-03-29T11:02:36.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 10 librados: watch_flush exit 2024-03-29T11:02:36.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T11:02:36.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 10 client.4664.objecter successfully canceled tick 2024-03-29T11:02:36.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 >> v1:172.21.15.183:6816/1412921548 conn(0x7fc88406dd20 legacy=0x7fc8840701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 10 monclient: shutdown 2024-03-29T11:02:36.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:36.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 >> v1:172.21.15.155:6790/0 conn(0x7fc8a413f650 legacy=0x7fc8a406ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:36.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 shutdown_connections 2024-03-29T11:02:36.539 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 >> 172.21.15.155:0/219019504 conn(0x7fc8a4066a00 msgr2=0x7fc8a40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:36.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 shutdown_connections 2024-03-29T11:02:36.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.559+0000 7fc8a92ae640 1 -- 172.21.15.155:0/219019504 wait complete. 2024-03-29T11:02:36.540 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:36.560+0000 7fc8a92ae640 1 librados: shutdown 2024-03-29T11:02:36.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:43.363 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:43.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.525+0000 7fc4e98ba640 10 monclient: get_monmap_and_config 2024-03-29T11:02:43.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.525+0000 7fc4e98ba640 10 monclient: build_initial_monmap 2024-03-29T11:02:43.504 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.525+0000 7fc4e98ba640 10 monclient: monmap: 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:02:43.525873+0000 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:02:43.525873+0000 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:02:43.505 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:43.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.526+0000 7fc4e98ba640 1 Processor -- start 2024-03-29T11:02:43.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.526+0000 7fc4e98ba640 1 -- start start 2024-03-29T11:02:43.506 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.526+0000 7fc4e98ba640 10 monclient: init 2024-03-29T11:02:43.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:43.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:43.507 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc0389e0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc038dc0 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc030660 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc0c54c0 con 0x7fc4dc030660 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc036430 con 0x7fc4dc038dc0 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc039840 con 0x7fc4dc0389e0 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.527+0000 7fc4e98ba640 10 monclient(hunting): authenticate will time out at 1466.819244s 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.528+0000 7fc4e88b8640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc4dc0389e0 0x7fc4dc033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:44736/0 (socket says 172.21.15.155:44736) 2024-03-29T11:02:43.508 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.528+0000 7fc4e88b8640 1 -- 172.21.15.155:0/3255753815 learned_addr learned my addr 172.21.15.155:0/3255753815 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:43.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.530+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379213009 0 0) 0x7fc4dc039840 con 0x7fc4dc0389e0 2024-03-29T11:02:43.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.530+0000 7fc4e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:43.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.530+0000 7fc4e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:43.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.530+0000 7fc4e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:43.509 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.530+0000 7fc4e37fe640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:02:43.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.530+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4cc003520 con 0x7fc4dc0389e0 2024-03-29T11:02:43.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 723927344 0 0) 0x7fc4cc003520 con 0x7fc4dc0389e0 2024-03-29T11:02:43.510 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 >> v1:172.21.15.183:6789/0 conn(0x7fc4dc038dc0 legacy=0x7fc4dc0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:43.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 >> v1:172.21.15.155:6789/0 conn(0x7fc4dc030660 legacy=0x7fc4dc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:43.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:43.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 1 monclient: found mon.noname-b 2024-03-29T11:02:43.512 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.155:6790/0 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc035690 con 0x7fc4dc0389e0 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc4e404f730 con 0x7fc4dc0389e0 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e98ba640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e98ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.531+0000 7fc4e37fe640 10 monclient: dump: 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.513 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e98ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 <== mon.2 v1:172.21.15.155:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4e40008d0 con 0x7fc4dc0389e0 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e98ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e98ba640 4 set_mon_vals no callback set 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e37fe640 1 -- 172.21.15.155:0/3255753815 <== mon.2 v1:172.21.15.155:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc4e40758b0 con 0x7fc4dc0389e0 2024-03-29T11:02:43.514 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.532+0000 7fc4e37fe640 10 monclient: dump: 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.533+0000 7fc4e98ba640 10 monclient: shutdown 2024-03-29T11:02:43.515 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.533+0000 7fc4e98ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:43.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.533+0000 7fc4e98ba640 1 -- 172.21.15.155:0/3255753815 >> v1:172.21.15.155:6790/0 conn(0x7fc4dc0389e0 legacy=0x7fc4dc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:43.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.533+0000 7fc4e98ba640 1 -- 172.21.15.155:0/3255753815 shutdown_connections 2024-03-29T11:02:43.516 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.533+0000 7fc4e98ba640 1 -- 172.21.15.155:0/3255753815 >> 172.21.15.155:0/3255753815 conn(0x7fc4dc005c00 msgr2=0x7fc4dc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.535+0000 7fc4e98ba640 1 -- 172.21.15.155:0/3255753815 shutdown_connections 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.535+0000 7fc4e98ba640 1 -- 172.21.15.155:0/3255753815 wait complete. 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.536+0000 7fc4e98ba640 10 monclient: build_initial_monmap 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.536+0000 7fc4e98ba640 10 monclient: monmap: 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:02:43.517 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:02:43.518 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:43.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.537+0000 7fc4e98ba640 1 librados: starting msgr at 2024-03-29T11:02:43.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.537+0000 7fc4e98ba640 1 librados: starting objecter 2024-03-29T11:02:43.518 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.537+0000 7fc4e98ba640 1 Processor -- start 2024-03-29T11:02:43.522 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.537+0000 7fc4e98ba640 1 -- start start 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.537+0000 7fc4e98ba640 1 librados: setting wanted keys 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.537+0000 7fc4e98ba640 1 librados: calling monclient init 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.537+0000 7fc4e98ba640 10 monclient: init 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc030660 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc02fed0 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0097b0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc0c3410 con 0x7fc4dc02fed0 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc02f9b0 con 0x7fc4dc030660 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc02f130 con 0x7fc4dc0097b0 2024-03-29T11:02:43.523 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e98ba640 10 monclient(hunting): authenticate will time out at 1466.830625s 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e90b9640 1 --1- >> v1:172.21.15.155:6790/0 conn(0x7fc4dc0097b0 0x7fc4dc0c2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.155:6790/0 says I am v1:172.21.15.155:45036/0 (socket says 172.21.15.155:45036) 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.538+0000 7fc4e90b9640 1 -- 172.21.15.155:0/748102387 learned_addr learned my addr 172.21.15.155:0/748102387 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 10 client.?.objecter ms_handle_connect 0x7fc4dc0097b0 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379557558 0 0) 0x7fc4dc02f130 con 0x7fc4dc0097b0 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d0004ee0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226048783 0 0) 0x7fc4d0004ee0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 --> v1:172.21.15.155:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4dc02f130 con 0x7fc4dc0097b0 2024-03-29T11:02:43.524 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.539+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc4d40018a0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.540+0000 7fc4e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.540+0000 7fc4e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.540+0000 7fc4e17fa640 10 monclient(hunting): dump: 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.540+0000 7fc4e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:43.525 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.540+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4292262290 0 0) 0x7fc4dc02f130 con 0x7fc4dc0097b0 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.540+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 >> v1:172.21.15.183:6789/0 conn(0x7fc4dc030660 legacy=0x7fc4dc1647e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.542+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 >> v1:172.21.15.155:6789/0 conn(0x7fc4dc02fed0 legacy=0x7fc4dc0c25f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 1 monclient: found mon.c 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 --> v1:172.21.15.155:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc00d7d0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 10 monclient: _finish_auth 0 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:02:43.526 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:43.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:02:43.528 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 10 monclient: _renew_subs 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 --> v1:172.21.15.155:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc164ef0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 10 monclient: _renew_subs 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 --> v1:172.21.15.155:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc0c8410 con 0x7fc4dc0097b0 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.543+0000 7fc4e98ba640 1 librados: init done 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.545+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4d4001a70 con 0x7fc4dc0097b0 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.545+0000 7fc4e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.545+0000 7fc4e2ffd640 4 set_mon_vals no callback set 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.546+0000 7fc4e98ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.547+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fc4d40038a0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.529 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.547+0000 7fc4e98ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.547+0000 7fc4e98ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.547+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c4005080 con 0x7fc4dc0097b0 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.547+0000 7fc4e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.547+0000 7fc4e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.547+0000 7fc4e17fa640 10 monclient: dump: 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:43.530 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:43.531 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:43.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.549+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fc4d4003a00 con 0x7fc4dc0097b0 2024-03-29T11:02:43.531 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.550+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7fc4d40849e0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.550+0000 7fc4e17fa640 10 client.4670.objecter ms_dispatch 0x7fc4e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:43.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.550+0000 7fc4e88b8640 10 monclient: get_auth_request con 0x7fc4d006dcc0 auth_method 0 2024-03-29T11:02:43.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.550+0000 7fc4e17fa640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:43.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.550+0000 7fc4e17fa640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:43.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.550+0000 7fc4e17fa640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T11:02:43.532 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.551+0000 7fc4e17fa640 10 client.4670.objecter ms_handle_connect 0x7fc4d006dcc0 2024-03-29T11:02:43.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.559+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4d40500f0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.559+0000 7fc4e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:43.538 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.559+0000 7fc4e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:43.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.890+0000 7fc4e98ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:43.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.890+0000 7fc4e98ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:43.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.890+0000 7fc4e98ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.155:6790/0 2024-03-29T11:02:43.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.890+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 --> v1:172.21.15.155:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4c4005330 con 0x7fc4dc0097b0 2024-03-29T11:02:43.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.891+0000 7fc4e17fa640 1 -- 172.21.15.155:0/748102387 <== mon.2 v1:172.21.15.155:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7fc4d40521f0 con 0x7fc4dc0097b0 2024-03-29T11:02:43.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.892+0000 7fc4e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:43.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.892+0000 7fc4e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:43.871 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:02:43.871 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:43.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.895+0000 7fc4e98ba640 10 librados: watch_flush enter 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 10 librados: watch_flush exit 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 10 client.4670.objecter successfully canceled tick 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 >> v1:172.21.15.183:6816/1412921548 conn(0x7fc4d006dcc0 legacy=0x7fc4d0070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 10 monclient: shutdown 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 >> v1:172.21.15.155:6790/0 conn(0x7fc4dc0097b0 legacy=0x7fc4dc0c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 shutdown_connections 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 >> 172.21.15.155:0/748102387 conn(0x7fc4dc005c00 msgr2=0x7fc4dc034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:43.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 shutdown_connections 2024-03-29T11:02:43.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 1 -- 172.21.15.155:0/748102387 wait complete. 2024-03-29T11:02:43.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:43.896+0000 7fc4e98ba640 1 librados: shutdown 2024-03-29T11:02:43.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:50.702 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.872+0000 7fb491855640 10 monclient: get_monmap_and_config 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.872+0000 7fb491855640 10 monclient: build_initial_monmap 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.872+0000 7fb491855640 10 monclient: monmap: 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T11:02:50.873323+0000 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T11:02:50.873323+0000 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-29T11:02:50.852 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-b 2024-03-29T11:02:50.853 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:50.853 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.873+0000 7fb491855640 1 Processor -- start 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.873+0000 7fb491855640 1 -- start start 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.873+0000 7fb491855640 10 monclient: init 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 10 monclient(hunting): picked mon.noname-b con 0x7fb48c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 10 monclient(hunting): picked mon.noname-c con 0x7fb48c13f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 10 monclient(hunting): picked mon.noname-a con 0x7fb48c0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c0aaef0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c145bd0 con 0x7fb48c13f650 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c145eb0 con 0x7fb48c1457f0 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:50.854 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.874+0000 7fb491855640 10 monclient(hunting): authenticate will time out at 1474.166809s 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb48b7fe640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb48c0a9d70 0x7fb48c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:57070/0 (socket says 172.21.15.155:57070) 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb48b7fe640 1 -- 172.21.15.155:0/3019608237 learned_addr learned my addr 172.21.15.155:0/3019608237 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770796862 0 0) 0x7fb48c0aaef0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb489ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb489ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb489ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb489ffb640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:02:50.858 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.875+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb474003520 con 0x7fb48c0a9d70 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170139943 0 0) 0x7fb474003520 con 0x7fb48c0a9d70 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 >> v1:172.21.15.155:6790/0 conn(0x7fb48c1457f0 legacy=0x7fb48c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 >> v1:172.21.15.183:6789/0 conn(0x7fb48c13f650 legacy=0x7fb48c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 1 monclient: found mon.noname-a 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.155:6789/0 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c146340 con 0x7fb48c0a9d70 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb484001690 con 0x7fb48c0a9d70 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:50.859 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb489ffb640 10 monclient: dump: 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb491855640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.876+0000 7fb491855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 <== mon.0 v1:172.21.15.155:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4840017f0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.860 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb489ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb489ffb640 1 -- 172.21.15.155:0/3019608237 <== mon.0 v1:172.21.15.155:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb484001c00 con 0x7fb48c0a9d70 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb489ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb489ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb489ffb640 10 monclient: dump: 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:50.861 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb491855640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.877+0000 7fb491855640 4 set_mon_vals no callback set 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.878+0000 7fb491855640 10 monclient: shutdown 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.878+0000 7fb491855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.878+0000 7fb491855640 1 -- 172.21.15.155:0/3019608237 >> v1:172.21.15.155:6789/0 conn(0x7fb48c0a9d70 legacy=0x7fb48c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.878+0000 7fb491855640 1 -- 172.21.15.155:0/3019608237 shutdown_connections 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.878+0000 7fb491855640 1 -- 172.21.15.155:0/3019608237 >> 172.21.15.155:0/3019608237 conn(0x7fb48c066a00 msgr2=0x7fb48c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.878+0000 7fb491855640 1 -- 172.21.15.155:0/3019608237 shutdown_connections 2024-03-29T11:02:50.862 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.878+0000 7fb491855640 1 -- 172.21.15.155:0/3019608237 wait complete. 2024-03-29T11:02:50.869 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.880+0000 7fb491855640 10 monclient: build_initial_monmap 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.880+0000 7fb491855640 10 monclient: monmap: 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.noname-c 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:50.870 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.880+0000 7fb491855640 1 librados: starting msgr at 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.880+0000 7fb491855640 1 librados: starting objecter 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.881+0000 7fb491855640 1 Processor -- start 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.881+0000 7fb491855640 1 -- start start 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.881+0000 7fb491855640 1 librados: setting wanted keys 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.881+0000 7fb491855640 1 librados: calling monclient init 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.881+0000 7fb491855640 10 monclient: init 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 10 monclient(hunting): picked mon.noname-a con 0x7fb48c0a9d70 addr v1:172.21.15.155:6789/0 2024-03-29T11:02:50.871 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 10 monclient(hunting): picked mon.noname-b con 0x7fb48c13f650 addr v1:172.21.15.183:6789/0 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 10 monclient(hunting): picked mon.noname-c con 0x7fb48c1457f0 addr v1:172.21.15.155:6790/0 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c1391e0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c13aef0 con 0x7fb48c13f650 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 1 -- --> v1:172.21.15.155:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c13b1d0 con 0x7fb48c1457f0 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb491855640 10 monclient(hunting): authenticate will time out at 1474.174563s 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb48a7fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb48c13f650 0x7fb48c140780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.155:49816/0 (socket says 172.21.15.155:49816) 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb48a7fc640 1 -- 172.21.15.155:0/1667131666 learned_addr learned my addr 172.21.15.155:0/1667131666 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.882+0000 7fb473fff640 10 client.?.objecter ms_handle_connect 0x7fb48c13f650 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:50.872 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 client.?.objecter ms_handle_connect 0x7fb48c0a9d70 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247260406 0 0) 0x7fb48c13aef0 con 0x7fb48c13f650 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb478004eb0 con 0x7fb48c13f650 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 client.?.objecter ms_handle_connect 0x7fb48c1457f0 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073323109 0 0) 0x7fb48c1391e0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:50.873 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c13aef0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1553236595 0 0) 0x7fb478004eb0 con 0x7fb48c13f650 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.883+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48c1391e0 con 0x7fb48c13f650 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb480001a30 con 0x7fb48c13f650 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): dump: 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.874 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 538771553 0 0) 0x7fb48c13aef0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.155:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb478004eb0 con 0x7fb48c0a9d70 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.2 v1:172.21.15.155:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128020026 0 0) 0x7fb48c13b1d0 con 0x7fb48c1457f0 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:50.875 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.155:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c13aef0 con 0x7fb48c1457f0 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb47c001b00 con 0x7fb48c0a9d70 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient(hunting): dump: 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:50.876 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 892969641 0 0) 0x7fb48c1391e0 con 0x7fb48c13f650 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 >> v1:172.21.15.155:6790/0 conn(0x7fb48c1457f0 legacy=0x7fb48c1e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 >> v1:172.21.15.155:6789/0 conn(0x7fb48c0a9d70 legacy=0x7fb48c139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 monclient: found mon.b 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c138e70 con 0x7fb48c13f650 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient: _finish_auth 0 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb473fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:50.877 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 10 monclient: _renew_subs 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c13a220 con 0x7fb48c13f650 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 10 monclient: _renew_subs 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c13bdd0 con 0x7fb48c13f650 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.884+0000 7fb491855640 1 librados: init done 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.885+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb480001b90 con 0x7fb48c13f650 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.885+0000 7fb473fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.885+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3352906891 0 0) 0x7fb480003a60 con 0x7fb48c13f650 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.885+0000 7fb473fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:50.878 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.885+0000 7fb4897fa640 4 set_mon_vals no callback set 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.885+0000 7fb473fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.885+0000 7fb473fff640 10 monclient: dump: 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:fsid 880eb1aa-520d-47ca-bf3e-0bcfe36d81b7 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:created 2024-03-29T10:51:14.989795+0000 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.a 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:2: v1:172.21.15.155:6790/0 mon.c 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr: 2024-03-29T11:02:50.879 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.886+0000 7fb491855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.886+0000 7fb491855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.886+0000 7fb491855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.886+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c005080 con 0x7fb48c13f650 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.888+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 69524592 0 0) 0x7fb480003bc0 con 0x7fb48c13f650 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.889+0000 7fb48affd640 10 monclient: get_auth_request con 0x7fb47806df70 auth_method 0 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.889+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1128280333 0 0) 0x7fb480084aa0 con 0x7fb48c13f650 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.889+0000 7fb473fff640 10 client.4546.objecter ms_dispatch 0x7fb48c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.889+0000 7fb473fff640 3 client.4546.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:02:50.880 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.889+0000 7fb473fff640 3 client.4546.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:02:50.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.889+0000 7fb473fff640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:02:50.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.889+0000 7fb473fff640 10 client.4546.objecter ms_handle_connect 0x7fb47806df70 2024-03-29T11:02:50.881 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.897+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4800503e0 con 0x7fb48c13f650 2024-03-29T11:02:50.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.897+0000 7fb473fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:50.882 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:50.897+0000 7fb473fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:51.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.267+0000 7fb491855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:51.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.267+0000 7fb491855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:51.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.267+0000 7fb491855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-29T11:02:51.247 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.267+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb46c005330 con 0x7fb48c13f650 2024-03-29T11:02:51.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.269+0000 7fb473fff640 1 -- 172.21.15.155:0/1667131666 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 441458623) 0x7fb480052310 con 0x7fb48c13f650 2024-03-29T11:02:51.248 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.269+0000 7fb473fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:51.249 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.269+0000 7fb473fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:51.249 INFO:teuthology.misc.health.smithi155.stdout: 2024-03-29T11:02:51.249 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":11,"fsid":"880eb1aa-520d-47ca-bf3e-0bcfe36d81b7","created":"2024-03-29T10:51:53.142841+0000","modified":"2024-03-29T11:01:58.081221+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:01:58.081221+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f08abf9e-cefc-4532-9ddd-4b175fc54b03","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b673f42b-b29e-4994-bc55-f4ed0a62e5c7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"dbb3193a-5420-4a34-8814-4a79c888bd62","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9e754e8b-aaf9-4153-bc61-27aed626f0d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2d805188-e83a-4e5c-99b1-cedd6690d4f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2b3d0c1-7dde-4bf7-927f-28af2dc16fb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3dad4236-bfc3-4e2e-b438-95ae2e8aa5e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c9a1875d-6ecd-4408-ae84-597c695ec1af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:51.252 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.272+0000 7fb491855640 10 librados: watch_flush enter 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 10 librados: watch_flush exit 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 >> v1:172.21.15.183:6816/1412921548 conn(0x7fb47806df70 legacy=0x7fb478070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 10 monclient: shutdown 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 >> v1:172.21.15.183:6789/0 conn(0x7fb48c13f650 legacy=0x7fb48c140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 shutdown_connections 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 >> 172.21.15.155:0/1667131666 conn(0x7fb48c066a00 msgr2=0x7fb48c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 shutdown_connections 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 1 -- 172.21.15.155:0/1667131666 wait complete. 2024-03-29T11:02:51.253 INFO:teuthology.misc.health.smithi155.stderr:2024-03-29T11:02:51.273+0000 7fb491855640 1 librados: shutdown 2024-03-29T11:02:51.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:51.267 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:02:51.269 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:02:51.269 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:02:51.269 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.303 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:02:51.303 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:02:51.303 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.373 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:02:51.373 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:02:51.373 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.450 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:02:51.450 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:02:51.450 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.527 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:02:51.528 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:02:51.528 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.573 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:02:51.574 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:02:51.574 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.607 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:02:51.607 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:02:51.608 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.681 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:02:51.682 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:02:51.682 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.750 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:02:51.751 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:02:51.751 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:02:51.751 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.775 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:02:51.775 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:02:51.775 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.815 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:02:51.815 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:02:51.816 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:02:51.816 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.827 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:02:51.827 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:02:51.827 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.878 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:02:51.879 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:02:51.879 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:02:51.889 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:02:51.890 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:02:51.890 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:02:51.961 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:02:51.961 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:02:54.145 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:02:54.146 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:02:54.265 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:02:54.266 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:02:54.323 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:02:54.324 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:02:54.398 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:02:54.399 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:02:57.860 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:02:57.860 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:02:57.922 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:02:57.922 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:02:57.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:02:57.987 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:02:58.046 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:02:58.048 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/data/mon.a.tgz 2024-03-29T11:02:58.048 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-29T11:02:58.067 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.SVW4GrCFQp 2024-03-29T11:02:58.067 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.SVW4GrCFQp 2024-03-29T11:02:58.244 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.SVW4GrCFQp 2024-03-29T11:02:58.333 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.SVW4GrCFQp is 126KB 2024-03-29T11:02:58.384 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.SVW4GrCFQp 2024-03-29T11:02:58.399 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/data/mon.c.tgz 2024-03-29T11:02:58.400 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-29T11:02:58.454 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.X20OpOL7w4 2024-03-29T11:02:58.455 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.X20OpOL7w4 2024-03-29T11:02:58.543 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.X20OpOL7w4 2024-03-29T11:02:58.629 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.X20OpOL7w4 is 126KB 2024-03-29T11:02:58.679 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.X20OpOL7w4 2024-03-29T11:02:58.693 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/data/mon.b.tgz 2024-03-29T11:02:58.693 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-29T11:02:58.711 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.IdLl6g7rLw 2024-03-29T11:02:58.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.IdLl6g7rLw 2024-03-29T11:02:58.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.IdLl6g7rLw 2024-03-29T11:02:58.910 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.IdLl6g7rLw is 126KB 2024-03-29T11:02:58.961 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.IdLl6g7rLw 2024-03-29T11:02:58.976 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:02:58.976 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:02:58.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:02:59.058 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:02:59.098 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:02:59.099 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:02:59.101 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/remote/smithi155/crash 2024-03-29T11:02:59.101 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:02:59.128 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/remote/smithi183/crash 2024-03-29T11:02:59.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:02:59.165 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:02:59.165 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:02:59.172 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:02:59.196 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-03-29T11:02:59.238 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39425.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41452.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38918.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39425.log.gz 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41032.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38918.log.gz 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39815.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39893.log 2024-03-29T11:02:59.239 INFO:teuthology.orchestra.run.smithi155.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39815.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39815.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39893.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40168.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38526.log 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38526.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-03-29T11:02:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39308.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38762.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38526.log.gz 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38762.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39308.log.gz 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38292.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39386.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38762.log.gz 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38879.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38292.log.gz 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39269.log 2024-03-29T11:02:59.241 INFO:teuthology.orchestra.run.smithi155.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39386.log.gz 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38331.log 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38879.log.gz 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39269.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38370.log 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39269.log.gz 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39152.log 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38331.log.gz 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38370.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gz 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41569.log 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38370.log.gz 2024-03-29T11:02:59.242 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39152.log.gz 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41569.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39932.log 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41569.log.gz 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40954.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39932.log.gz 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39113.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2024-03-29T11:02:59.243 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41491.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 85.4%/var/log/ceph/ceph-client.admin.41296.log: -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39464.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35082.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40010.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39464.log.gz 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39854.log: /var/log/ceph/ceph-client.admin.35082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41256.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40010.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-03-29T11:02:59.244 INFO:teuthology.orchestra.run.smithi155.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35082.log.gz 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40010.log.gz 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.1.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41256.log.gz 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38840.log 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40480.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38840.log.gz 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40718.log 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40718.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39347.log 2024-03-29T11:02:59.245 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40562.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40718.log.gz 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38801.log 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40679.log 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39347.log.gz 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38801.log: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz -5 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40679.log: gzip 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38801.log.gz -5 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40441.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40679.log.gz 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:02:59.246 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39191.log 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41215.log 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39542.log 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39191.log.gz 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38606.log 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41215.log.gz 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40088.log: 87.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.39542.log.gz 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39581.log 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34688.log 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-03-29T11:02:59.247 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41608.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38606.log.gz 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.34688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39581.log.gz 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38448.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41608.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34688.log.gz 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40993.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41608.log.gz 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mgr.y.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38448.log.gz 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41335.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38409.log: /var/log/ceph/ceph-client.admin.40285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-03-29T11:02:59.248 INFO:teuthology.orchestra.run.smithi155.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41335.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39230.log 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40757.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41335.log.gz 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38723.log 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39230.log: gzip -5 --verbose 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr: -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38723.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39230.log.gz 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40835.log 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38723.log.gz 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39035.log 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41530.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38957.log 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40835.log.gz 2024-03-29T11:02:59.249 INFO:teuthology.orchestra.run.smithi155.stderr: 85.7%/var/log/ceph/ceph-client.admin.39035.log: -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39074.log 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39035.log.gz 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38645.log 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39074.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38957.log.gz 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39074.log.gz 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40207.log: /var/log/ceph/ceph-client.admin.38645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38645.log.gz 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:02:59.250 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.2.log: 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34979.log 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40640.log 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.34979.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38487.log 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34979.log.gz 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40640.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40640.log.gz 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39503.log 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38684.log 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz/var/log/ceph/ceph-client.admin.39503.log: 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38487.log.gz 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39503.log.gz 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-03-29T11:02:59.251 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38567.log 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40129.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38684.log.gz 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38567.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38567.log.gz 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40519.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38996.log 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39620.log 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.38996.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28869.log 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39620.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38996.log.gz 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-03-29T11:02:59.252 INFO:teuthology.orchestra.run.smithi155.stderr: 85.9%/var/log/ceph/ceph.tmp-client.admin.28869.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28869.log.gz 2024-03-29T11:02:59.253 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-03-29T11:02:59.253 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39620.log.gz 2024-03-29T11:02:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39971.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:02:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41374.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-03-29T11:02:59.253 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mon.c.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-03-29T11:02:59.258 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:02:59.258 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34521.log 2024-03-29T11:02:59.258 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34641.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34481.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34441.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34521.log.gz 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34481.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34641.log.gz 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34441.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34481.log.gz 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34441.log.gz 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34561.log 2024-03-29T11:02:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34681.log 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34721.log 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34681.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34561.log.gz 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr: -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.34721.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34681.log.gz 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mgr.x.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34721.log.gz 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34601.log 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34601.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:02:59.260 INFO:teuthology.orchestra.run.smithi183.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34601.log.gz 2024-03-29T11:02:59.286 INFO:teuthology.orchestra.run.smithi183.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:03:10.819 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:03:10.994 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:03:11.123 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:03:11.372 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:03:11.443 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:03:14.895 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:03:15.089 INFO:teuthology.orchestra.run.smithi155.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:03:15.502 INFO:teuthology.orchestra.run.smithi183.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:03:50.978 INFO:teuthology.orchestra.run.smithi155.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:03:51.819 INFO:teuthology.orchestra.run.smithi155.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:03:51.821 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-29T11:03:51.821 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m52.637s 2024-03-29T11:03:51.821 INFO:teuthology.orchestra.run.smithi155.stderr:user 2m33.439s 2024-03-29T11:03:51.821 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m5.592s 2024-03-29T11:04:10.790 INFO:teuthology.orchestra.run.smithi183.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:04:10.792 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-29T11:04:10.792 INFO:teuthology.orchestra.run.smithi183.stderr:real 1m11.569s 2024-03-29T11:04:10.792 INFO:teuthology.orchestra.run.smithi183.stderr:user 1m59.327s 2024-03-29T11:04:10.792 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m4.466s 2024-03-29T11:04:10.792 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:04:10.793 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/remote/smithi155/log 2024-03-29T11:04:10.794 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:04:32.862 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/remote/smithi183/log 2024-03-29T11:04:32.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:04:51.173 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:04:51.378 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c58d47d726df494ba371cff54b542431 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:04:51.381 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:04:51.389 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:04:51.396 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:04:51.396 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:04:51.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:04:51.489 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:04:51.497 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:04:51.497 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:04:51.500 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:04:51.513 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:04:51.516 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:04:51.519 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:04:51.519 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-29T11:04:51.519 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +403us[ +423us] +/- 66ms 2024-03-29T11:04:51.520 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -201us[ -182us] +/- 61ms 2024-03-29T11:04:51.520 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 65 -206us[ -186us] +/- 92ms 2024-03-29T11:04:51.520 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:04:51.521 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:04:51.522 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-29T11:04:51.522 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +396us[ +408us] +/- 66ms 2024-03-29T11:04:51.522 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -199us[ -186us] +/- 61ms 2024-03-29T11:04:51.522 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -211us[ -199us] +/- 92ms 2024-03-29T11:04:51.522 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:04:51.523 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:04:51.531 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:04:51.532 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:04:51.539 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:04:51.628 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:04:51.699 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:04:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:51.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:04:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:51.762 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:04:51.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:04:51.778 INFO:teuthology.task.internal:Duration was 1233.918539 seconds 2024-03-29T11:04:51.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:04:51.787 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:04:51.787 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:04:51.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:04:51.831 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:04:51.847 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:04:52.101 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:04:52.101 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-03-29T11:04:52.101 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:04:52.121 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-29T11:04:52.121 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:04:52.144 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:04:52.144 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:04:52.163 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:04:52.213 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:04:52.214 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:04:52.231 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:04:52.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:04:52.367 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:04:52.368 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:04:52.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:04:52.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:04:52.431 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:52.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:52.460 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-03-29T11:04:52.487 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-29T11:04:52.513 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:52.536 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:04:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:04:52.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:04:52.577 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:04:52.578 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/remote/smithi155 2024-03-29T11:04:52.578 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:04:52.614 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658/remote/smithi183 2024-03-29T11:04:52.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:04:52.649 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:04:52.649 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:04:52.652 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:04:52.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:04:52.710 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:04:52.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:04:52.720 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:04:52.721 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:04:52.723 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:04:52.735 INFO:teuthology.orchestra.run.smithi155.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:04 /home/ubuntu/cephtest 2024-03-29T11:04:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:04 /home/ubuntu/cephtest 2024-03-29T11:04:52.755 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:04:52.764 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:04:52.783 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:04:52.806 DEBUG:teuthology.nuke:shortname: smithi155 2024-03-29T11:04:52.807 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:04:52.831 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:35:09.256737', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEtSkJut390BqVDfRKg4OAHfgPKyf9sIPkl6ioaxhFCEc4NbCyuccSf5xRUNVAYJB5c98YMumFwqGiBz6H09lQk='} 2024-03-29T11:04:52.834 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-29T11:04:52.834 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:04:52.857 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622658', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:35:09.258771', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOni4yEAR1WulQ0qK868whKpIbAuHVwVcRlsWDswMtE+hdMc1ichRhUwpUyRfbVf9GAPksxrWMZORAoAu2/LjRM='} 2024-03-29T11:04:52.886 INFO:teuthology.orchestra.console.smithi155:Power off 2024-03-29T11:04:52.887 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:04:52.912 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-03-29T11:04:52.913 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:52.929 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-29T11:04:52.929 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:04:52.939 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-29T11:04:52.952 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-29T11:04:52.953 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:52.976 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-29T11:04:56.941 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:56.977 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:04:57.052 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-29T11:04:57.087 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-29T11:05:01.054 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:01.090 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:01.166 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-29T11:05:01.202 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-29T11:05:05.170 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:05.203 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:05:05.281 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-03-29T11:05:05.314 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-29T11:05:05.382 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-03-29T11:05:05.415 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-29T11:05:05.520 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1233.918538570404 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c58d47d726df494ba371cff54b542431 status: fail success: false 2024-03-29T11:05:05.520 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:05:05.600 INFO:teuthology.run:FAIL